2025-10-10T01:22:00.5290041Z Current runner version: '2.328.0' 2025-10-10T01:22:00.5295981Z Runner name: 'i-0f89836b6c30bbee2' 2025-10-10T01:22:00.5297124Z Runner group name: 'default' 2025-10-10T01:22:00.5297995Z Machine name: 'ip-10-0-14-165' 2025-10-10T01:22:00.5300789Z ##[group]GITHUB_TOKEN Permissions 2025-10-10T01:22:00.5302811Z Contents: read 2025-10-10T01:22:00.5303593Z Metadata: read 2025-10-10T01:22:00.5304084Z ##[endgroup] 2025-10-10T01:22:00.5306495Z Secret source: Actions 2025-10-10T01:22:00.5307171Z Prepare workflow directory 2025-10-10T01:22:00.5775620Z Prepare all required actions 2025-10-10T01:22:00.5809092Z Getting action download info 2025-10-10T01:22:00.9043411Z Download action repository 'pytorch/test-infra@main' (SHA:264eed5d70b428e3aa5c1a7c98e4330f866e183f) 2025-10-10T01:22:03.3471812Z Download action repository 'pytorch/pytorch@main' (SHA:c7b57d93490d83c9cf59908eb34754882664a1c2) 2025-10-10T01:22:19.3786437Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-10-10T01:22:19.7735229Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-10-10T01:22:20.0212852Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-10-10T01:22:20.2427098Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-10-10T01:22:20.5363299Z Getting action download info 2025-10-10T01:22:20.6754499Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-10-10T01:22:20.9453810Z Getting action download info 2025-10-10T01:22:21.0778537Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-10-10T01:22:21.3325071Z Getting action download info 2025-10-10T01:22:21.4776528Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-10-10T01:22:21.9219985Z Getting action download info 2025-10-10T01:22:22.0515467Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (344e6365a0068c2d2847fcec0c55dd53291d475e) 2025-10-10T01:22:22.0519031Z ##[group] Inputs 2025-10-10T01:22:22.0519430Z build-environment: linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T01:22:22.0524494Z test-matrix: {"include": [{"config": "vllm_basic_correctness_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_basic_models_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_entrypoints_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_regression_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_processor_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_pytorch_compilation_unit_tests", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_languagde_model_test_extended_generation_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_distributed_test_2_gpu_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 0, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 1, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 2, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 3, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_tp_test_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}, {"config": "vllm_distributed_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}]} 2025-10-10T01:22:22.0529945Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:22:22.0530567Z sync-tag: 2025-10-10T01:22:22.0531266Z timeout-minutes: 240 2025-10-10T01:22:22.0531469Z use-gha: 2025-10-10T01:22:22.0531648Z dashboard-tag: 2025-10-10T01:22:22.0531840Z s3-bucket: gha-artifacts 2025-10-10T01:22:22.0532041Z aws-role-to-assume: 2025-10-10T01:22:22.0532540Z disable-monitor: false 2025-10-10T01:22:22.0532796Z monitor-log-interval: 5 2025-10-10T01:22:22.0533029Z monitor-data-collect-interval: 1 2025-10-10T01:22:22.0533273Z ##[endgroup] 2025-10-10T01:22:22.0533635Z Complete job name: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:22:22.1207321Z A job started hook has been configured by the self-hosted runner administrator 2025-10-10T01:22:22.1306140Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-10-10T01:22:22.1316014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:22:22.1316556Z ##[endgroup] 2025-10-10T01:22:23.4237416Z Runner Type: linux.g6.4xlarge.experimental.nvidia.gpu 2025-10-10T01:22:23.4237903Z Instance Type: g6.4xlarge 2025-10-10T01:22:23.4238106Z AMI Name: unknown 2025-10-10T01:22:23.4280913Z AMI ID: ami-08982f1c5bf93d976 2025-10-10T01:22:28.3231315Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-10-10T01:22:28.3231677Z with: 2025-10-10T01:22:28.3232217Z github-secret: *** 2025-10-10T01:22:28.3232753Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-10-10T01:22:28.3233302Z activate-with-label: false 2025-10-10T01:22:28.3233499Z label: with-ssh 2025-10-10T01:22:28.3233678Z remove-existing-keys: true 2025-10-10T01:22:28.3233876Z fail-silently: true 2025-10-10T01:22:28.3234047Z env: 2025-10-10T01:22:28.3234194Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:22:28.3234398Z ##[endgroup] 2025-10-10T01:22:28.4530217Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-10-10T01:22:28.4531594Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-10-10T01:22:28.4710137Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-10-10T01:22:28.4710504Z with: 2025-10-10T01:22:28.4710669Z no-sudo: true 2025-10-10T01:22:28.4710977Z submodules: recursive 2025-10-10T01:22:28.4711319Z fetch-depth: 0 2025-10-10T01:22:28.4711550Z env: 2025-10-10T01:22:28.4711771Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:22:28.4712040Z ##[endgroup] 2025-10-10T01:22:28.4791240Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T01:22:28.4791962Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T01:22:28.4806370Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:22:28.4806663Z env: 2025-10-10T01:22:28.4806862Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:22:28.4807104Z ##[endgroup] 2025-10-10T01:22:28.4905229Z ##[group]Run # Use all available CPUs for fetching 2025-10-10T01:22:28.4905576Z # Use all available CPUs for fetching 2025-10-10T01:22:28.4905834Z cd "${GITHUB_WORKSPACE}" 2025-10-10T01:22:28.4906083Z git config --global fetch.parallel 0 2025-10-10T01:22:28.4906556Z git config --global submodule.fetchJobs 0 2025-10-10T01:22:28.4906798Z  2025-10-10T01:22:28.4907124Z # Clean workspace. The default checkout action should also do this, but 2025-10-10T01:22:28.4907497Z # do it here as well just in case 2025-10-10T01:22:28.4907731Z if [[ -d .git ]]; then 2025-10-10T01:22:28.4907947Z  if [ -z "${NO_SUDO}" ]; then 2025-10-10T01:22:28.4908167Z  sudo git clean -ffdx 2025-10-10T01:22:28.4908366Z  else 2025-10-10T01:22:28.4908539Z  git clean -ffdx 2025-10-10T01:22:28.4908736Z  fi 2025-10-10T01:22:28.4908890Z fi 2025-10-10T01:22:28.4916225Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:22:28.4916524Z env: 2025-10-10T01:22:28.4916688Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:22:28.4916880Z NO_SUDO: true 2025-10-10T01:22:28.4917040Z ##[endgroup] 2025-10-10T01:22:28.5068522Z ##[group]Run actions/checkout@v4 2025-10-10T01:22:28.5068757Z with: 2025-10-10T01:22:28.5068951Z ref: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:22:28.5069191Z fetch-depth: 0 2025-10-10T01:22:28.5069368Z submodules: recursive 2025-10-10T01:22:28.5069561Z show-progress: false 2025-10-10T01:22:28.5069768Z repository: pytorch/pytorch 2025-10-10T01:22:28.5070086Z token: *** 2025-10-10T01:22:28.5070261Z ssh-strict: true 2025-10-10T01:22:28.5070431Z ssh-user: git 2025-10-10T01:22:28.5070616Z persist-credentials: true 2025-10-10T01:22:28.5070816Z clean: true 2025-10-10T01:22:28.5071001Z sparse-checkout-cone-mode: true 2025-10-10T01:22:28.5071218Z fetch-tags: false 2025-10-10T01:22:28.5071390Z lfs: false 2025-10-10T01:22:28.5071551Z set-safe-directory: true 2025-10-10T01:22:28.5071751Z env: 2025-10-10T01:22:28.5071914Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:22:28.5072103Z ##[endgroup] 2025-10-10T01:22:28.6054094Z Syncing repository: pytorch/pytorch 2025-10-10T01:22:28.6055312Z ##[group]Getting Git version info 2025-10-10T01:22:28.6055706Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-10-10T01:22:28.6056208Z [command]/usr/bin/git version 2025-10-10T01:22:28.6271035Z git version 2.50.1 2025-10-10T01:22:28.6297099Z ##[endgroup] 2025-10-10T01:22:28.6306767Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/50609c45-58d8-485c-9a33-08373d5a3f19/.gitconfig' 2025-10-10T01:22:28.6326120Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/50609c45-58d8-485c-9a33-08373d5a3f19' before making global git config changes 2025-10-10T01:22:28.6326838Z Adding repository directory to the temporary git global config as a safe directory 2025-10-10T01:22:28.6330967Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T01:22:28.6388853Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-10-10T01:22:28.6392034Z ##[group]Initializing the repository 2025-10-10T01:22:28.6395516Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T01:22:28.6460437Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-10-10T01:22:28.6460924Z hint: is subject to change. To configure the initial branch name to use in all 2025-10-10T01:22:28.6461353Z hint: of your new repositories, which will suppress this warning, call: 2025-10-10T01:22:28.6461663Z hint: 2025-10-10T01:22:28.6461904Z hint: git config --global init.defaultBranch 2025-10-10T01:22:28.6462165Z hint: 2025-10-10T01:22:28.6462413Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-10-10T01:22:28.6462827Z hint: 'development'. The just-created branch can be renamed via this command: 2025-10-10T01:22:28.6463137Z hint: 2025-10-10T01:22:28.6463307Z hint: git branch -m 2025-10-10T01:22:28.6463500Z hint: 2025-10-10T01:22:28.6463754Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-10-10T01:22:28.6470069Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-10-10T01:22:28.6478513Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-10-10T01:22:28.6520217Z ##[endgroup] 2025-10-10T01:22:28.6520567Z ##[group]Disabling automatic garbage collection 2025-10-10T01:22:28.6523454Z [command]/usr/bin/git config --local gc.auto 0 2025-10-10T01:22:28.6552950Z ##[endgroup] 2025-10-10T01:22:28.6553330Z ##[group]Setting up auth 2025-10-10T01:22:28.6558736Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-10-10T01:22:28.6589171Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-10-10T01:22:28.6971956Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-10-10T01:22:28.7002553Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-10-10T01:22:28.7372086Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-10-10T01:22:28.7428449Z ##[endgroup] 2025-10-10T01:22:28.7428901Z ##[group]Fetching the repository 2025-10-10T01:22:28.7435625Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-10-10T01:23:14.0070333Z From https://github.com/pytorch/pytorch 2025-10-10T01:23:14.0070917Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-10-10T01:23:14.0073849Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-10-10T01:23:14.0074711Z * [new branch] BootcampDynamo -> origin/BootcampDynamo 2025-10-10T01:23:14.0075219Z * [new branch] DynamoFixGit -> origin/DynamoFixGit 2025-10-10T01:23:14.0075848Z * [new branch] DynamoVariaT -> origin/DynamoVariaT 2025-10-10T01:23:14.0078500Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-10-10T01:23:14.0080632Z * [new branch] ISSUE-154849 -> origin/ISSUE-154849 2025-10-10T01:23:14.0083950Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-10-10T01:23:14.0085759Z * [new branch] IvanKobzarev/stack/2 -> origin/IvanKobzarev/stack/2 2025-10-10T01:23:14.0088063Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-10-10T01:23:14.0089602Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-10-10T01:23:14.0091534Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-10-10T01:23:14.0093281Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-10-10T01:23:14.0095010Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-10-10T01:23:14.0097256Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-10-10T01:23:14.0099054Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-10-10T01:23:14.0101268Z * [new branch] VLA_exp -> origin/VLA_exp 2025-10-10T01:23:14.0103160Z * [new branch] actually-run-mps-aot-inductor -> origin/actually-run-mps-aot-inductor 2025-10-10T01:23:14.0104954Z * [new branch] add_compile_benchmarking -> origin/add_compile_benchmarking 2025-10-10T01:23:14.0106715Z * [new branch] add_op_tests -> origin/add_op_tests 2025-10-10T01:23:14.0108539Z * [new branch] add_op_to_dashboard -> origin/add_op_to_dashboard 2025-10-10T01:23:14.0110357Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-10-10T01:23:14.0112165Z * [new branch] addremovefunction -> origin/addremovefunction 2025-10-10T01:23:14.0113962Z * [new branch] addvllmtest -> origin/addvllmtest 2025-10-10T01:23:14.0116492Z * [new branch] adi/test -> origin/adi/test 2025-10-10T01:23:14.0118239Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-10-10T01:23:14.0120061Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-10-10T01:23:14.0121760Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-10-10T01:23:14.0123978Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-10-10T01:23:14.0125256Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-10-10T01:23:14.0126986Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-10-10T01:23:14.0129064Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-10-10T01:23:14.0132134Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-10-10T01:23:14.0133895Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-10-10T01:23:14.0135702Z * [new branch] alt-disable -> origin/alt-disable 2025-10-10T01:23:14.0138245Z * [new branch] angelayi/allow_fake -> origin/angelayi/allow_fake 2025-10-10T01:23:14.0140080Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-10-10T01:23:14.0141924Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-10-10T01:23:14.0143621Z * [new branch] angelayi/benchmark2 -> origin/angelayi/benchmark2 2025-10-10T01:23:14.0145338Z * [new branch] angelayi/benchmark3 -> origin/angelayi/benchmark3 2025-10-10T01:23:14.0147135Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-10-10T01:23:14.0148785Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-10-10T01:23:14.0150841Z * [new branch] angelayi/customop -> origin/angelayi/customop 2025-10-10T01:23:14.0153014Z * [new branch] angelayi/fix_mps -> origin/angelayi/fix_mps 2025-10-10T01:23:14.0155131Z * [new branch] angelayi/lint -> origin/angelayi/lint 2025-10-10T01:23:14.0157230Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-10-10T01:23:14.0158944Z * [new branch] angelayi/opaque_obj_v2 -> origin/angelayi/opaque_obj_v2 2025-10-10T01:23:14.0160900Z * [new branch] angelayi/pattern -> origin/angelayi/pattern 2025-10-10T01:23:14.0162764Z * [new branch] angelayi/pattern_in_out_2 -> origin/angelayi/pattern_in_out_2 2025-10-10T01:23:14.0164488Z * [new branch] angelayi/post_grad -> origin/angelayi/post_grad 2025-10-10T01:23:14.0166380Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-10-10T01:23:14.0168179Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-10-10T01:23:14.0170013Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-10-10T01:23:14.0171778Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-10-10T01:23:14.0173564Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-10-10T01:23:14.0175338Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-10-10T01:23:14.0177118Z * [new branch] angelayi/wrap_grad -> origin/angelayi/wrap_grad 2025-10-10T01:23:14.0178821Z * [new branch] annotate_1 -> origin/annotate_1 2025-10-10T01:23:14.0180705Z * [new branch] annotation_bw -> origin/annotation_bw 2025-10-10T01:23:14.0182407Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-10-10T01:23:14.0184221Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-10-10T01:23:14.0186062Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-10-10T01:23:14.0187792Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-10-10T01:23:14.0189532Z * [new branch] aoti_metal_shimify -> origin/aoti_metal_shimify 2025-10-10T01:23:14.0191253Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-10-10T01:23:14.0193018Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-10-10T01:23:14.0194784Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-10-10T01:23:14.0196837Z * [new branch] aoti_windows_mingw -> origin/aoti_windows_mingw 2025-10-10T01:23:14.0198668Z * [new branch] aoti_windows_mingw_2 -> origin/aoti_windows_mingw_2 2025-10-10T01:23:14.0202001Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-10-10T01:23:14.0203652Z * [new branch] async_tp -> origin/async_tp 2025-10-10T01:23:14.0205616Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-10-10T01:23:14.0207450Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-10-10T01:23:14.0209148Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-10-10T01:23:14.0210916Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-10-10T01:23:14.0212851Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-10-10T01:23:14.0214621Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-10-10T01:23:14.0216427Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-10-10T01:23:14.0218860Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-10-10T01:23:14.0220857Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-10-10T01:23:14.0222594Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-10-10T01:23:14.0224578Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-10-10T01:23:14.0226277Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-10-10T01:23:14.0228094Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-10-10T01:23:14.0229796Z * [new branch] attention_benchmark -> origin/attention_benchmark 2025-10-10T01:23:14.0231763Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-10-10T01:23:14.0233448Z * [new branch] b200_op_bench -> origin/b200_op_bench 2025-10-10T01:23:14.0235809Z * [new branch] bahuang/annotation -> origin/bahuang/annotation 2025-10-10T01:23:14.0237551Z * [new branch] bahuang/debug_mode -> origin/bahuang/debug_mode 2025-10-10T01:23:14.0239313Z * [new branch] bahuang/debug_mode_default -> origin/bahuang/debug_mode_default 2025-10-10T01:23:14.0241062Z * [new branch] bahuang/debug_mode_fix -> origin/bahuang/debug_mode_fix 2025-10-10T01:23:14.0242607Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-10-10T01:23:14.0244449Z * [new branch] bahuang/dt_reduce_mean -> origin/bahuang/dt_reduce_mean 2025-10-10T01:23:14.0246181Z * [new branch] bahuang/dtensor_demo -> origin/bahuang/dtensor_demo 2025-10-10T01:23:14.0248205Z * [new branch] bahuang/export_dtensor -> origin/bahuang/export_dtensor 2025-10-10T01:23:14.0250362Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-10-10T01:23:14.0252243Z * [new branch] bahuang/fix_debug_mode2 -> origin/bahuang/fix_debug_mode2 2025-10-10T01:23:14.0254057Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-10-10T01:23:14.0255882Z * [new branch] bahuang/noop_redistribute -> origin/bahuang/noop_redistribute 2025-10-10T01:23:14.0257729Z * [new branch] bahuang/reland -> origin/bahuang/reland 2025-10-10T01:23:14.0259520Z * [new branch] bahuang/reland_fake_export -> origin/bahuang/reland_fake_export 2025-10-10T01:23:14.0261221Z * [new branch] bahuang/rename -> origin/bahuang/rename 2025-10-10T01:23:14.0263081Z * [new branch] bahuang/test -> origin/bahuang/test 2025-10-10T01:23:14.0265652Z * [new branch] base/1.5 -> origin/base/1.5 2025-10-10T01:23:14.0267576Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-10-10T01:23:14.0269311Z * [new branch] bc-lint-test-new-config -> origin/bc-lint-test-new-config 2025-10-10T01:23:14.0271198Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-10-10T01:23:14.0273019Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-10-10T01:23:14.0275427Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-10-10T01:23:14.0277790Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-10-10T01:23:14.0280415Z * [new branch] bf/cg-custom-wrapper -> origin/bf/cg-custom-wrapper 2025-10-10T01:23:14.0282118Z * [new branch] bf/cg-error-re-record -> origin/bf/cg-error-re-record 2025-10-10T01:23:14.0283941Z * [new branch] bf/cg-partition-custom-op-mutation -> origin/bf/cg-partition-custom-op-mutation 2025-10-10T01:23:14.0285506Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-10-10T01:23:14.0287218Z * [new branch] bf/cg-warn-dynamic-shapes -> origin/bf/cg-warn-dynamic-shapes 2025-10-10T01:23:14.0289742Z * [new branch] bf/cherry-pick-partition-share-default-device-context -> origin/bf/cherry-pick-partition-share-default-device-context 2025-10-10T01:23:14.0291010Z * [new branch] bf/clean-hf -> origin/bf/clean-hf 2025-10-10T01:23:14.0293364Z * [new branch] bf/clean-timm -> origin/bf/clean-timm 2025-10-10T01:23:14.0294847Z * [new branch] bf/clean-torchbench -> origin/bf/clean-torchbench 2025-10-10T01:23:14.0297111Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-10-10T01:23:14.0298801Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-10-10T01:23:14.0300614Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-10-10T01:23:14.0302593Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-10-10T01:23:14.0304081Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-10-10T01:23:14.0305943Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-10-10T01:23:14.0307804Z * [new branch] bf/minor-cg-config-doc -> origin/bf/minor-cg-config-doc 2025-10-10T01:23:14.0309593Z * [new branch] bf/minor-fa-tma-config -> origin/bf/minor-fa-tma-config 2025-10-10T01:23:14.0311371Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-10-10T01:23:14.0313114Z * [new branch] bf/partition-custom-op-alias -> origin/bf/partition-custom-op-alias 2025-10-10T01:23:14.0314944Z * [new branch] bf/partition-default-device-context -> origin/bf/partition-default-device-context 2025-10-10T01:23:14.0316683Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-10-10T01:23:14.0318495Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-10-10T01:23:14.0320361Z * [new branch] bf/rope -> origin/bf/rope 2025-10-10T01:23:14.0322260Z * [new branch] bf16_support -> origin/bf16_support 2025-10-10T01:23:14.0324194Z * [new branch] bf16_support_per_channel -> origin/bf16_support_per_channel 2025-10-10T01:23:14.0326100Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-10-10T01:23:14.0327960Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-10-10T01:23:14.0329692Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-10-10T01:23:14.0331466Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-10-10T01:23:14.0333144Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-10-10T01:23:14.0334933Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-10-10T01:23:14.0336592Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-10-10T01:23:14.0338302Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-10-10T01:23:14.0340082Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-10-10T01:23:14.0341695Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-10-10T01:23:14.0343408Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-10-10T01:23:14.0345126Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-10-10T01:23:14.0347022Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-10-10T01:23:14.0348718Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-10-10T01:23:14.0350437Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-10-10T01:23:14.0352207Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-10-10T01:23:14.0354632Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-10-10T01:23:14.0357142Z * [new branch] brister/break_scatter_src_is_tensor -> origin/brister/break_scatter_src_is_tensor 2025-10-10T01:23:14.0358880Z * [new branch] brister/fx_cond -> origin/brister/fx_cond 2025-10-10T01:23:14.0360810Z * [new branch] brister/fx_dynamic_input -> origin/brister/fx_dynamic_input 2025-10-10T01:23:14.0362398Z * [new branch] brister/fx_index_put -> origin/brister/fx_index_put 2025-10-10T01:23:14.0364161Z * [new branch] brister/fx_no_python_slow -> origin/brister/fx_no_python_slow 2025-10-10T01:23:14.0365781Z * [new branch] brister/fx_scatter_reduce -> origin/brister/fx_scatter_reduce 2025-10-10T01:23:14.0367812Z * [new branch] brister/fx_unbacked_symbols -> origin/brister/fx_unbacked_symbols 2025-10-10T01:23:14.0369976Z * [new branch] brister/property_type_check -> origin/brister/property_type_check 2025-10-10T01:23:14.0371743Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-10-10T01:23:14.0373610Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-10-10T01:23:14.0375280Z * [new branch] build-aarch64-wheels -> origin/build-aarch64-wheels 2025-10-10T01:23:14.0377071Z * [new branch] bwd-backup -> origin/bwd-backup 2025-10-10T01:23:14.0378914Z * [new branch] c57382a49 -> origin/c57382a49 2025-10-10T01:23:14.0380635Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-10-10T01:23:14.0382322Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-10-10T01:23:14.0384999Z * [new branch] camyll/cherrypick_0098e5636d3afa7c75aef8c447a5c402ea9ed524 -> origin/camyll/cherrypick_0098e5636d3afa7c75aef8c447a5c402ea9ed524 2025-10-10T01:23:14.0386923Z * [new branch] camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 -> origin/camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 2025-10-10T01:23:14.0389290Z * [new branch] camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 2025-10-10T01:23:14.0390885Z * [new branch] camyll/revert_5d749ceb92c2c28bcfbdf918b4ab99b1a91fcb50 -> origin/camyll/revert_5d749ceb92c2c28bcfbdf918b4ab99b1a91fcb50 2025-10-10T01:23:14.0393435Z * [new branch] camyllh/cherrypick_5e7be988003a38be49227cfaa9bff6a2ea9e6929_v2 -> origin/camyllh/cherrypick_5e7be988003a38be49227cfaa9bff6a2ea9e6929_v2 2025-10-10T01:23:14.0394991Z * [new branch] camyllh/cherrypick_dda071587f0522a16b237f92cbe27fd13a1a1c11 -> origin/camyllh/cherrypick_dda071587f0522a16b237f92cbe27fd13a1a1c11 2025-10-10T01:23:14.0397843Z * [new branch] camyllh/release2_9_cherrypick/dda071587f0522a16b237f92cbe27fd13a1a1c11 -> origin/camyllh/release2_9_cherrypick/dda071587f0522a16b237f92cbe27fd13a1a1c11 2025-10-10T01:23:14.0399783Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-10-10T01:23:14.0401757Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0403680Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0405319Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0407105Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0408947Z * [new branch] cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0410816Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0412693Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0414538Z * [new branch] cherry-pick-161299-by-pytorch_bot_bot_ -> origin/cherry-pick-161299-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0416335Z * [new branch] cherry-pick-161394-by-pytorch_bot_bot_ -> origin/cherry-pick-161394-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0418120Z * [new branch] cherry-pick-161430-by-pytorch_bot_bot_ -> origin/cherry-pick-161430-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0420021Z * [new branch] cherry-pick-162168-by-pytorch_bot_bot_ -> origin/cherry-pick-162168-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0421975Z * [new branch] cherry-pick-162194-by-pytorch_bot_bot_ -> origin/cherry-pick-162194-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0423865Z * [new branch] cherry-pick-162240-by-pytorch_bot_bot_ -> origin/cherry-pick-162240-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0426013Z * [new branch] cherry-pick-162295-by-pytorch_bot_bot_ -> origin/cherry-pick-162295-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0427821Z * [new branch] cherry-pick-162323-by-pytorch_bot_bot_ -> origin/cherry-pick-162323-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0429540Z * [new branch] cherry-pick-162425-by-pytorch_bot_bot_ -> origin/cherry-pick-162425-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0431546Z * [new branch] cherry-pick-162530-by-pytorch_bot_bot_ -> origin/cherry-pick-162530-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0433355Z * [new branch] cherry-pick-162555-by-pytorch_bot_bot_ -> origin/cherry-pick-162555-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0435876Z * [new branch] cherry-pick-162566-by-pytorch_bot_bot_ -> origin/cherry-pick-162566-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0437637Z * [new branch] cherry-pick-162587-by-pytorch_bot_bot_ -> origin/cherry-pick-162587-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0439506Z * [new branch] cherry-pick-162622-by-pytorch_bot_bot_ -> origin/cherry-pick-162622-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0441443Z * [new branch] cherry-pick-162657-by-pytorch_bot_bot_ -> origin/cherry-pick-162657-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0443201Z * [new branch] cherry-pick-162680-by-pytorch_bot_bot_ -> origin/cherry-pick-162680-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0445089Z * [new branch] cherry-pick-162693-by-pytorch_bot_bot_ -> origin/cherry-pick-162693-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0446872Z * [new branch] cherry-pick-162744-by-pytorch_bot_bot_ -> origin/cherry-pick-162744-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0448788Z * [new branch] cherry-pick-162764-by-pytorch_bot_bot_ -> origin/cherry-pick-162764-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0450715Z * [new branch] cherry-pick-162865-by-pytorch_bot_bot_ -> origin/cherry-pick-162865-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0452484Z * [new branch] cherry-pick-162866-by-pytorch_bot_bot_ -> origin/cherry-pick-162866-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0454333Z * [new branch] cherry-pick-162877-by-pytorch_bot_bot_ -> origin/cherry-pick-162877-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0456180Z * [new branch] cherry-pick-162950-by-pytorch_bot_bot_ -> origin/cherry-pick-162950-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0457848Z * [new branch] cherry-pick-163008-by-pytorch_bot_bot_ -> origin/cherry-pick-163008-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0459781Z * [new branch] cherry-pick-163111-by-pytorch_bot_bot_ -> origin/cherry-pick-163111-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0461570Z * [new branch] cherry-pick-163112-by-pytorch_bot_bot_ -> origin/cherry-pick-163112-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0463324Z * [new branch] cherry-pick-163152-by-pytorch_bot_bot_ -> origin/cherry-pick-163152-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0465054Z * [new branch] cherry-pick-163171-by-pytorch_bot_bot_ -> origin/cherry-pick-163171-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0466855Z * [new branch] cherry-pick-163194-by-pytorch_bot_bot_ -> origin/cherry-pick-163194-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0468745Z * [new branch] cherry-pick-163227-by-pytorch_bot_bot_ -> origin/cherry-pick-163227-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0470490Z * [new branch] cherry-pick-163269-by-pytorch_bot_bot_ -> origin/cherry-pick-163269-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0472219Z * [new branch] cherry-pick-163298-by-pytorch_bot_bot_ -> origin/cherry-pick-163298-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0474029Z * [new branch] cherry-pick-163315-by-pytorch_bot_bot_ -> origin/cherry-pick-163315-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0475822Z * [new branch] cherry-pick-163339-by-pytorch_bot_bot_ -> origin/cherry-pick-163339-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0477536Z * [new branch] cherry-pick-163341-by-pytorch_bot_bot_ -> origin/cherry-pick-163341-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0479373Z * [new branch] cherry-pick-163370-by-pytorch_bot_bot_ -> origin/cherry-pick-163370-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0481363Z * [new branch] cherry-pick-163383-by-pytorch_bot_bot_ -> origin/cherry-pick-163383-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0483078Z * [new branch] cherry-pick-163426-by-pytorch_bot_bot_ -> origin/cherry-pick-163426-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0485397Z * [new branch] cherry-pick-163549-by-pytorch_bot_bot_ -> origin/cherry-pick-163549-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0487287Z * [new branch] cherry-pick-163571-by-pytorch_bot_bot_ -> origin/cherry-pick-163571-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0489051Z * [new branch] cherry-pick-163578-by-pytorch_bot_bot_ -> origin/cherry-pick-163578-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0490835Z * [new branch] cherry-pick-163581-by-pytorch_bot_bot_ -> origin/cherry-pick-163581-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0492636Z * [new branch] cherry-pick-163585-by-pytorch_bot_bot_ -> origin/cherry-pick-163585-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0494385Z * [new branch] cherry-pick-163587-by-pytorch_bot_bot_ -> origin/cherry-pick-163587-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0496412Z * [new branch] cherry-pick-163598-by-pytorch_bot_bot_ -> origin/cherry-pick-163598-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0498199Z * [new branch] cherry-pick-163661-by-pytorch_bot_bot_ -> origin/cherry-pick-163661-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0500019Z * [new branch] cherry-pick-163677-by-pytorch_bot_bot_ -> origin/cherry-pick-163677-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0501705Z * [new branch] cherry-pick-163682-by-pytorch_bot_bot_ -> origin/cherry-pick-163682-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0503423Z * [new branch] cherry-pick-163712-by-pytorch_bot_bot_ -> origin/cherry-pick-163712-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0505175Z * [new branch] cherry-pick-163719-by-pytorch_bot_bot_ -> origin/cherry-pick-163719-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0507122Z * [new branch] cherry-pick-163768-by-pytorch_bot_bot_ -> origin/cherry-pick-163768-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0508861Z * [new branch] cherry-pick-163776-by-pytorch_bot_bot_ -> origin/cherry-pick-163776-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0510645Z * [new branch] cherry-pick-163797-by-pytorch_bot_bot_ -> origin/cherry-pick-163797-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0512453Z * [new branch] cherry-pick-163837-by-pytorch_bot_bot_ -> origin/cherry-pick-163837-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0514181Z * [new branch] cherry-pick-163886-by-pytorch_bot_bot_ -> origin/cherry-pick-163886-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0515946Z * [new branch] cherry-pick-163903-by-pytorch_bot_bot_ -> origin/cherry-pick-163903-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0517875Z * [new branch] cherry-pick-163956-by-pytorch_bot_bot_ -> origin/cherry-pick-163956-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0519809Z * [new branch] cherry-pick-163988-by-pytorch_bot_bot_ -> origin/cherry-pick-163988-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0521641Z * [new branch] cherry-pick-164093-by-pytorch_bot_bot_ -> origin/cherry-pick-164093-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0523463Z * [new branch] cherry-pick-164108-by-pytorch_bot_bot_ -> origin/cherry-pick-164108-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0525286Z * [new branch] cherry-pick-164138-by-pytorch_bot_bot_ -> origin/cherry-pick-164138-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0526890Z * [new branch] cherry-pick-164190 -> origin/cherry-pick-164190 2025-10-10T01:23:14.0528829Z * [new branch] cherry-pick-164470-by-pytorch_bot_bot_ -> origin/cherry-pick-164470-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0531015Z * [new branch] cherry-pick-164575-by-pytorch_bot_bot_ -> origin/cherry-pick-164575-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0533022Z * [new branch] cherry-pick-164774-by-pytorch_bot_bot_ -> origin/cherry-pick-164774-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0535011Z * [new branch] cherry-pick-164870-by-pytorch_bot_bot_ -> origin/cherry-pick-164870-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0536774Z * [new branch] cherry-pick-164946-by-pytorch_bot_bot_ -> origin/cherry-pick-164946-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0538594Z * [new branch] cherry-pick-165013-by-pytorch_bot_bot_ -> origin/cherry-pick-165013-by-pytorch_bot_bot_ 2025-10-10T01:23:14.0540313Z * [new branch] cherry_pick_graph_custom -> origin/cherry_pick_graph_custom 2025-10-10T01:23:14.0542094Z * [new branch] cherrypick-e88cca0691 -> origin/cherrypick-e88cca0691 2025-10-10T01:23:14.0543966Z * [new branch] chuanqi129-patch-1 -> origin/chuanqi129-patch-1 2025-10-10T01:23:14.0545676Z * [new branch] ck_dlpack -> origin/ck_dlpack 2025-10-10T01:23:14.0547426Z * [new branch] codegen_trace -> origin/codegen_trace 2025-10-10T01:23:14.0549354Z * [new branch] codex-testing -> origin/codex-testing 2025-10-10T01:23:14.0552369Z * [new branch] codex/add-metadata-field-for-file-path -> origin/codex/add-metadata-field-for-file-path 2025-10-10T01:23:14.0554154Z * [new branch] codex/add-test-for-inductor-local-cache-behavior -> origin/codex/add-test-for-inductor-local-cache-behavior 2025-10-10T01:23:14.0556250Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-10-10T01:23:14.0558342Z * [new branch] codex/enhance-cuda.matmul-with-allow_splitk-argument -> origin/codex/enhance-cuda.matmul-with-allow_splitk-argument 2025-10-10T01:23:14.0560108Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-10-10T01:23:14.0561922Z * [new branch] codex/refactor-dimension-handling-in-shape.cu -> origin/codex/refactor-dimension-handling-in-shape.cu 2025-10-10T01:23:14.0563813Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-10-10T01:23:14.0565462Z * [new branch] codex/remove-allow-untyped-defs-and-fix-type-errors -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors 2025-10-10T01:23:14.0567304Z * [new branch] codex/remove-allow-untyped-defs-and-fix-type-errors-vx0jek -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors-vx0jek 2025-10-10T01:23:14.0568720Z * [new branch] compile_kernel_include_dir -> origin/compile_kernel_include_dir 2025-10-10T01:23:14.0570735Z * [new branch] context_test -> origin/context_test 2025-10-10T01:23:14.0572500Z * [new branch] conv1d_decomp -> origin/conv1d_decomp 2025-10-10T01:23:14.0574238Z * [new branch] conv_autotune -> origin/conv_autotune 2025-10-10T01:23:14.0576710Z * [new branch] copilot/fix-157446 -> origin/copilot/fix-157446 2025-10-10T01:23:14.0578312Z * [new branch] copilot/fix-163730 -> origin/copilot/fix-163730 2025-10-10T01:23:14.0580887Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-10-10T01:23:14.0582754Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-10-10T01:23:14.0584559Z * [new branch] cpp_head -> origin/cpp_head 2025-10-10T01:23:14.0586383Z * [new branch] crcrpar-patch-1 -> origin/crcrpar-patch-1 2025-10-10T01:23:14.0588640Z * [new branch] csl/add_win_shard -> origin/csl/add_win_shard 2025-10-10T01:23:14.0590334Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-10-10T01:23:14.0592009Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-10-10T01:23:14.0593689Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-10-10T01:23:14.0595413Z * [new branch] csl/fix_internal_graph_executor -> origin/csl/fix_internal_graph_executor 2025-10-10T01:23:14.0597961Z * [new branch] csl/fix_nightly_docs_push -> origin/csl/fix_nightly_docs_push 2025-10-10T01:23:14.0600233Z * [new branch] csl/inductor_h100_nightly -> origin/csl/inductor_h100_nightly 2025-10-10T01:23:14.0601969Z * [new branch] csl/katex -> origin/csl/katex 2025-10-10T01:23:14.0603782Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-10-10T01:23:14.0605482Z * [new branch] csl/lint_no_submodules -> origin/csl/lint_no_submodules 2025-10-10T01:23:14.0607207Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-10-10T01:23:14.0608914Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-10-10T01:23:14.0610674Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-10-10T01:23:14.0612445Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-10-10T01:23:14.0614148Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-10-10T01:23:14.0615976Z * [new branch] csl/reuse_old_whl_fix_metadata -> origin/csl/reuse_old_whl_fix_metadata 2025-10-10T01:23:14.0617608Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-10-10T01:23:14.0619423Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-10-10T01:23:14.0621215Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-10-10T01:23:14.0622954Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-10-10T01:23:14.0624869Z * [new branch] csl/test_info_status -> origin/csl/test_info_status 2025-10-10T01:23:14.0626583Z * [new branch] csl/test_info_upload_changes -> origin/csl/test_info_upload_changes 2025-10-10T01:23:14.0628235Z * [new branch] csl/test_owners_ao_sparse -> origin/csl/test_owners_ao_sparse 2025-10-10T01:23:14.0630108Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-10-10T01:23:14.0631684Z * [new branch] csl/test_owners_cuda -> origin/csl/test_owners_cuda 2025-10-10T01:23:14.0633407Z * [new branch] csl/test_owners_distributed -> origin/csl/test_owners_distributed 2025-10-10T01:23:14.0635319Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-10-10T01:23:14.0636979Z * [new branch] csl/testing_better_job_name -> origin/csl/testing_better_job_name 2025-10-10T01:23:14.0638688Z * [new branch] csl/vllm_pin_labeler -> origin/csl/vllm_pin_labeler 2025-10-10T01:23:14.0640606Z * [new branch] csl/win_cpp_tests -> origin/csl/win_cpp_tests 2025-10-10T01:23:14.0642301Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-10-10T01:23:14.0644117Z * [new branch] cu_stream_api -> origin/cu_stream_api 2025-10-10T01:23:14.0645920Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-10-10T01:23:14.0647737Z * [new branch] cublasnowdeterministic -> origin/cublasnowdeterministic 2025-10-10T01:23:14.0649502Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-10-10T01:23:14.0651315Z * [new branch] cuda-include-paths-fix -> origin/cuda-include-paths-fix 2025-10-10T01:23:14.0653109Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-10-10T01:23:14.0655523Z * [new branch] d4l3k/delete_hook -> origin/d4l3k/delete_hook 2025-10-10T01:23:14.0657925Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-10-10T01:23:14.0659658Z * [new branch] dcp_zoc -> origin/dcp_zoc 2025-10-10T01:23:14.0661407Z * [new branch] debug-guard -> origin/debug-guard 2025-10-10T01:23:14.0663406Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-10-10T01:23:14.0668642Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-10-10T01:23:14.0670796Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-10-10T01:23:14.0672513Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-10-10T01:23:14.0675417Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-10-10T01:23:14.0677941Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-10-10T01:23:14.0679906Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-10-10T01:23:14.0682137Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-10-10T01:23:14.0683920Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-10-10T01:23:14.0685923Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-10-10T01:23:14.0687850Z * [new branch] dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg 2025-10-10T01:23:14.0689753Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-10-10T01:23:14.0691575Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-10-10T01:23:14.0693951Z * [new branch] dev/joona/topk_newapi -> origin/dev/joona/topk_newapi 2025-10-10T01:23:14.0696443Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-10-10T01:23:14.0698357Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-10-10T01:23:14.0700165Z * [new branch] disable -> origin/disable 2025-10-10T01:23:14.0701918Z * [new branch] disp_counter -> origin/disp_counter 2025-10-10T01:23:14.0703784Z * [new branch] dtensor-issues -> origin/dtensor-issues 2025-10-10T01:23:14.0705556Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-10-10T01:23:14.0707943Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-10-10T01:23:14.0709707Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-10-10T01:23:14.0711541Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-10-10T01:23:14.0713171Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-10-10T01:23:14.0715215Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-10-10T01:23:14.0717065Z * [new branch] enable-keep-going-for-trunk-tags -> origin/enable-keep-going-for-trunk-tags 2025-10-10T01:23:14.0718782Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-10-10T01:23:14.0720767Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-10-10T01:23:14.0723160Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-10-10T01:23:14.0724956Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-10-10T01:23:14.0726640Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-10-10T01:23:14.0728349Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-10-10T01:23:14.0730448Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-10-10T01:23:14.0732420Z * [new branch] exclamaforte/fix-exhuastive-autotuning-reland -> origin/exclamaforte/fix-exhuastive-autotuning-reland 2025-10-10T01:23:14.0734547Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-10-10T01:23:14.0736301Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-10-10T01:23:14.0738043Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-10-10T01:23:14.0739965Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-10-10T01:23:14.0742039Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-10-10T01:23:14.0743758Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-10-10T01:23:14.0745870Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-10-10T01:23:14.0747497Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-10-10T01:23:14.0749301Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-10-10T01:23:14.0751147Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-10-10T01:23:14.0752864Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-10-10T01:23:14.0754733Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-10-10T01:23:14.0756423Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-10-10T01:23:14.0758156Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-10-10T01:23:14.0760025Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-10-10T01:23:14.0762325Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-10-10T01:23:14.0764117Z * [new branch] exec -> origin/exec 2025-10-10T01:23:14.0766115Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-10-10T01:23:14.0767868Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-10-10T01:23:14.0769774Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-10-10T01:23:14.0771689Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-10-10T01:23:14.0773699Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-10-10T01:23:14.0775462Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-10-10T01:23:14.0777225Z * [new branch] export-D76885271 -> origin/export-D76885271 2025-10-10T01:23:14.0779041Z * [new branch] export-D76885620 -> origin/export-D76885620 2025-10-10T01:23:14.0780793Z * [new branch] export-D76936623 -> origin/export-D76936623 2025-10-10T01:23:14.0782631Z * [new branch] export-D76958268 -> origin/export-D76958268 2025-10-10T01:23:14.0784447Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-10-10T01:23:14.0786289Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-10-10T01:23:14.0788108Z * [new branch] export-D78580107 -> origin/export-D78580107 2025-10-10T01:23:14.0789908Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-10-10T01:23:14.0791919Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-10-10T01:23:14.0793652Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-10-10T01:23:14.0795451Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-10-10T01:23:14.0797542Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-10-10T01:23:14.0799461Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-10-10T01:23:14.0801403Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-10-10T01:23:14.0803168Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-10-10T01:23:14.0805032Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-10-10T01:23:14.0806812Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-10-10T01:23:14.0808539Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-10-10T01:23:14.0810307Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-10-10T01:23:14.0812072Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-10-10T01:23:14.0813898Z * [new branch] export-D79230339 -> origin/export-D79230339 2025-10-10T01:23:14.0815661Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-10-10T01:23:14.0817454Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-10-10T01:23:14.0819394Z * [new branch] export-D79378362 -> origin/export-D79378362 2025-10-10T01:23:14.0821124Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-10-10T01:23:14.0823353Z * [new branch] export-D80948073 -> origin/export-D80948073 2025-10-10T01:23:14.0825335Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-10-10T01:23:14.0827139Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-10-10T01:23:14.0828855Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-10-10T01:23:14.0830733Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-10-10T01:23:14.0832604Z * [new branch] export-D81651226 -> origin/export-D81651226 2025-10-10T01:23:14.0834375Z * [new branch] export-D81698719 -> origin/export-D81698719 2025-10-10T01:23:14.0836180Z * [new branch] export-D82140619 -> origin/export-D82140619 2025-10-10T01:23:14.0838069Z * [new branch] export-D82174075 -> origin/export-D82174075 2025-10-10T01:23:14.0840141Z * [new branch] export-D82232574 -> origin/export-D82232574 2025-10-10T01:23:14.0841966Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-10-10T01:23:14.0843752Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-10-10T01:23:14.0845647Z * [new branch] export-D82380307 -> origin/export-D82380307 2025-10-10T01:23:14.0847492Z * [new branch] export-D82597111 -> origin/export-D82597111 2025-10-10T01:23:14.0849384Z * [new branch] export-D83023706 -> origin/export-D83023706 2025-10-10T01:23:14.0852046Z * [new branch] export-D83195687 -> origin/export-D83195687 2025-10-10T01:23:14.0853845Z * [new branch] export-D83200714 -> origin/export-D83200714 2025-10-10T01:23:14.0855642Z * [new branch] export-D83378477 -> origin/export-D83378477 2025-10-10T01:23:14.0857464Z * [new branch] export-D83390563 -> origin/export-D83390563 2025-10-10T01:23:14.0859263Z * [new branch] export-D83390775 -> origin/export-D83390775 2025-10-10T01:23:14.0861091Z * [new branch] export-D83391942 -> origin/export-D83391942 2025-10-10T01:23:14.0862912Z * [new branch] export-D83395610 -> origin/export-D83395610 2025-10-10T01:23:14.0864690Z * [new branch] export-D83539263 -> origin/export-D83539263 2025-10-10T01:23:14.0866465Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-10-10T01:23:14.0868376Z * [new branch] export-D83591083 -> origin/export-D83591083 2025-10-10T01:23:14.0870366Z * [new branch] export-D83609850 -> origin/export-D83609850 2025-10-10T01:23:14.0872048Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-10-10T01:23:14.0873806Z * [new branch] export-D83714690 -> origin/export-D83714690 2025-10-10T01:23:14.0875732Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-10-10T01:23:14.0877600Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-10-10T01:23:14.0879477Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-10-10T01:23:14.0881464Z * [new branch] export-D84009392 -> origin/export-D84009392 2025-10-10T01:23:14.0883240Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-10-10T01:23:14.0885094Z * [new branch] export-D84098898 -> origin/export-D84098898 2025-10-10T01:23:14.0887074Z * [new branch] export-D84103213 -> origin/export-D84103213 2025-10-10T01:23:14.0888850Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-10-10T01:23:14.0890639Z * [new branch] export-reland -> origin/export-reland 2025-10-10T01:23:14.0892642Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-10-10T01:23:14.0894359Z * [new branch] extend_lift_up_op -> origin/extend_lift_up_op 2025-10-10T01:23:14.0896474Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-10-10T01:23:14.0898390Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-10-10T01:23:14.0900126Z * [new branch] ezyang-war -> origin/ezyang-war 2025-10-10T01:23:14.0902417Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-10-10T01:23:14.0904155Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-10-10T01:23:14.0906060Z * [new branch] fadeputr-fix-fbgemm_genai-build -> origin/fadeputr-fix-fbgemm_genai-build 2025-10-10T01:23:14.0908478Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-10-10T01:23:14.0910334Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-10-10T01:23:14.0912798Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-10-10T01:23:14.0914760Z * [new branch] fca -> origin/fca 2025-10-10T01:23:14.0916559Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-10-10T01:23:14.0918381Z * [new branch] fca5 -> origin/fca5 2025-10-10T01:23:14.0920938Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-10-10T01:23:14.0923075Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-10-10T01:23:14.0924847Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-10-10T01:23:14.0927260Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-10-10T01:23:14.0928925Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-10-10T01:23:14.0930613Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-10-10T01:23:14.0932306Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-10-10T01:23:14.0933946Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-10-10T01:23:14.0935589Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-10-10T01:23:14.0937218Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-10-10T01:23:14.0938900Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-10-10T01:23:14.0940639Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-10-10T01:23:14.0942401Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-10-10T01:23:14.0944190Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-10-10T01:23:14.0945982Z * [new branch] fix-fqn -> origin/fix-fqn 2025-10-10T01:23:14.0947932Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-10-10T01:23:14.0950163Z * [new branch] fix-upload-vllm-wheel-credential -> origin/fix-upload-vllm-wheel-credential 2025-10-10T01:23:14.0951934Z * [new branch] fix_153389 -> origin/fix_153389 2025-10-10T01:23:14.0953729Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-10-10T01:23:14.0955689Z * [new branch] fix_op_benchmark -> origin/fix_op_benchmark 2025-10-10T01:23:14.0957407Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-10-10T01:23:14.0959210Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-10-10T01:23:14.0961105Z * [new branch] fixes -> origin/fixes 2025-10-10T01:23:14.0963377Z * [new branch] fixes-triage -> origin/fixes-triage 2025-10-10T01:23:14.0965156Z * [new branch] fixflashgit -> origin/fixflashgit 2025-10-10T01:23:14.0966927Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-10-10T01:23:14.0968722Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-10-10T01:23:14.0970479Z * [new branch] flex-flash -> origin/flex-flash 2025-10-10T01:23:14.0972405Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-10-10T01:23:14.0974232Z * [new branch] flex_flash -> origin/flex_flash 2025-10-10T01:23:14.0976730Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-10-10T01:23:14.0978432Z * [new branch] free-stack2 -> origin/free-stack2 2025-10-10T01:23:14.0980758Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-10-10T01:23:14.0982229Z * [new branch] fsdpv2_3d -> origin/fsdpv2_3d 2025-10-10T01:23:14.0983922Z * [new branch] fsdpv2_3d_m1 -> origin/fsdpv2_3d_m1 2025-10-10T01:23:14.0985817Z * [new branch] fused_moving_avg_obs_fake_quant_half_support -> origin/fused_moving_avg_obs_fake_quant_half_support 2025-10-10T01:23:14.0987522Z * [new branch] fx_cpp -> origin/fx_cpp 2025-10-10T01:23:14.0990003Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-10-10T01:23:14.0993780Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-10-10T01:23:14.0995538Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-10-10T01:23:14.0999684Z * [new branch] gh/ColinPeppler/80/base -> origin/gh/ColinPeppler/80/base 2025-10-10T01:23:14.1001541Z * [new branch] gh/ColinPeppler/80/head -> origin/gh/ColinPeppler/80/head 2025-10-10T01:23:14.1003270Z * [new branch] gh/ColinPeppler/80/orig -> origin/gh/ColinPeppler/80/orig 2025-10-10T01:23:14.1005762Z * [new branch] gh/ColinPeppler/81/base -> origin/gh/ColinPeppler/81/base 2025-10-10T01:23:14.1007463Z * [new branch] gh/ColinPeppler/81/head -> origin/gh/ColinPeppler/81/head 2025-10-10T01:23:14.1009212Z * [new branch] gh/ColinPeppler/81/orig -> origin/gh/ColinPeppler/81/orig 2025-10-10T01:23:14.1011360Z * [new branch] gh/ColinPeppler/82/base -> origin/gh/ColinPeppler/82/base 2025-10-10T01:23:14.1013122Z * [new branch] gh/ColinPeppler/82/head -> origin/gh/ColinPeppler/82/head 2025-10-10T01:23:14.1014900Z * [new branch] gh/ColinPeppler/82/orig -> origin/gh/ColinPeppler/82/orig 2025-10-10T01:23:14.1017319Z * [new branch] gh/ColinPeppler/83/base -> origin/gh/ColinPeppler/83/base 2025-10-10T01:23:14.1019171Z * [new branch] gh/ColinPeppler/83/head -> origin/gh/ColinPeppler/83/head 2025-10-10T01:23:14.1020925Z * [new branch] gh/ColinPeppler/83/orig -> origin/gh/ColinPeppler/83/orig 2025-10-10T01:23:14.1023332Z * [new branch] gh/ColinPeppler/84/base -> origin/gh/ColinPeppler/84/base 2025-10-10T01:23:14.1025454Z * [new branch] gh/ColinPeppler/84/head -> origin/gh/ColinPeppler/84/head 2025-10-10T01:23:14.1027906Z * [new branch] gh/ColinPeppler/85/base -> origin/gh/ColinPeppler/85/base 2025-10-10T01:23:14.1029510Z * [new branch] gh/ColinPeppler/85/head -> origin/gh/ColinPeppler/85/head 2025-10-10T01:23:14.1031970Z * [new branch] gh/ColinPeppler/86/base -> origin/gh/ColinPeppler/86/base 2025-10-10T01:23:14.1033631Z * [new branch] gh/ColinPeppler/86/head -> origin/gh/ColinPeppler/86/head 2025-10-10T01:23:14.1035816Z * [new branch] gh/ColinPeppler/87/base -> origin/gh/ColinPeppler/87/base 2025-10-10T01:23:14.1037605Z * [new branch] gh/ColinPeppler/87/head -> origin/gh/ColinPeppler/87/head 2025-10-10T01:23:14.1039861Z * [new branch] gh/ColinPeppler/88/base -> origin/gh/ColinPeppler/88/base 2025-10-10T01:23:14.1041595Z * [new branch] gh/ColinPeppler/88/head -> origin/gh/ColinPeppler/88/head 2025-10-10T01:23:14.1043828Z * [new branch] gh/ColinPeppler/89/base -> origin/gh/ColinPeppler/89/base 2025-10-10T01:23:14.1045536Z * [new branch] gh/ColinPeppler/89/head -> origin/gh/ColinPeppler/89/head 2025-10-10T01:23:14.1047763Z * [new branch] gh/ColinPeppler/90/base -> origin/gh/ColinPeppler/90/base 2025-10-10T01:23:14.1049491Z * [new branch] gh/ColinPeppler/90/head -> origin/gh/ColinPeppler/90/head 2025-10-10T01:23:14.1051700Z * [new branch] gh/ColinPeppler/91/base -> origin/gh/ColinPeppler/91/base 2025-10-10T01:23:14.1053378Z * [new branch] gh/ColinPeppler/91/head -> origin/gh/ColinPeppler/91/head 2025-10-10T01:23:14.1055565Z * [new branch] gh/ColinPeppler/92/base -> origin/gh/ColinPeppler/92/base 2025-10-10T01:23:14.1057338Z * [new branch] gh/ColinPeppler/92/head -> origin/gh/ColinPeppler/92/head 2025-10-10T01:23:14.1059666Z * [new branch] gh/ColinPeppler/93/base -> origin/gh/ColinPeppler/93/base 2025-10-10T01:23:14.1061393Z * [new branch] gh/ColinPeppler/93/head -> origin/gh/ColinPeppler/93/head 2025-10-10T01:23:14.1063089Z * [new branch] gh/ColinPeppler/93/orig -> origin/gh/ColinPeppler/93/orig 2025-10-10T01:23:14.1065801Z * [new branch] gh/ColinPeppler/94/base -> origin/gh/ColinPeppler/94/base 2025-10-10T01:23:14.1067729Z * [new branch] gh/ColinPeppler/94/head -> origin/gh/ColinPeppler/94/head 2025-10-10T01:23:14.1069476Z * [new branch] gh/ColinPeppler/94/orig -> origin/gh/ColinPeppler/94/orig 2025-10-10T01:23:14.1071883Z * [new branch] gh/ColinPeppler/95/base -> origin/gh/ColinPeppler/95/base 2025-10-10T01:23:14.1073875Z * [new branch] gh/ColinPeppler/95/head -> origin/gh/ColinPeppler/95/head 2025-10-10T01:23:14.1075641Z * [new branch] gh/ColinPeppler/95/orig -> origin/gh/ColinPeppler/95/orig 2025-10-10T01:23:14.1078508Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-10-10T01:23:14.1080583Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-10-10T01:23:14.1083576Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-10-10T01:23:14.1085301Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-10-10T01:23:14.1088131Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-10-10T01:23:14.1089904Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-10-10T01:23:14.1091750Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-10-10T01:23:14.1094035Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-10-10T01:23:14.1095730Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-10-10T01:23:14.1097785Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-10-10T01:23:14.1100006Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-10-10T01:23:14.1101763Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-10-10T01:23:14.1103475Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-10-10T01:23:14.1105716Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-10-10T01:23:14.1107433Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-10-10T01:23:14.1109163Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-10-10T01:23:14.1111997Z * [new branch] gh/H-Huang/187/base -> origin/gh/H-Huang/187/base 2025-10-10T01:23:14.1113648Z * [new branch] gh/H-Huang/187/head -> origin/gh/H-Huang/187/head 2025-10-10T01:23:14.1115392Z * [new branch] gh/H-Huang/187/orig -> origin/gh/H-Huang/187/orig 2025-10-10T01:23:14.1117724Z * [new branch] gh/H-Huang/207/base -> origin/gh/H-Huang/207/base 2025-10-10T01:23:14.1119543Z * [new branch] gh/H-Huang/207/head -> origin/gh/H-Huang/207/head 2025-10-10T01:23:14.1121311Z * [new branch] gh/H-Huang/207/orig -> origin/gh/H-Huang/207/orig 2025-10-10T01:23:14.1123575Z * [new branch] gh/H-Huang/210/base -> origin/gh/H-Huang/210/base 2025-10-10T01:23:14.1125286Z * [new branch] gh/H-Huang/210/head -> origin/gh/H-Huang/210/head 2025-10-10T01:23:14.1126998Z * [new branch] gh/H-Huang/210/orig -> origin/gh/H-Huang/210/orig 2025-10-10T01:23:14.1129342Z * [new branch] gh/H-Huang/212/base -> origin/gh/H-Huang/212/base 2025-10-10T01:23:14.1131041Z * [new branch] gh/H-Huang/212/head -> origin/gh/H-Huang/212/head 2025-10-10T01:23:14.1132767Z * [new branch] gh/H-Huang/212/orig -> origin/gh/H-Huang/212/orig 2025-10-10T01:23:14.1135125Z * [new branch] gh/H-Huang/214/base -> origin/gh/H-Huang/214/base 2025-10-10T01:23:14.1136915Z * [new branch] gh/H-Huang/214/head -> origin/gh/H-Huang/214/head 2025-10-10T01:23:14.1138682Z * [new branch] gh/H-Huang/214/orig -> origin/gh/H-Huang/214/orig 2025-10-10T01:23:14.1140875Z * [new branch] gh/H-Huang/215/base -> origin/gh/H-Huang/215/base 2025-10-10T01:23:14.1142584Z * [new branch] gh/H-Huang/215/head -> origin/gh/H-Huang/215/head 2025-10-10T01:23:14.1144245Z * [new branch] gh/H-Huang/215/orig -> origin/gh/H-Huang/215/orig 2025-10-10T01:23:14.1146592Z * [new branch] gh/H-Huang/216/base -> origin/gh/H-Huang/216/base 2025-10-10T01:23:14.1148341Z * [new branch] gh/H-Huang/216/head -> origin/gh/H-Huang/216/head 2025-10-10T01:23:14.1150095Z * [new branch] gh/H-Huang/216/orig -> origin/gh/H-Huang/216/orig 2025-10-10T01:23:14.1152440Z * [new branch] gh/H-Huang/217/base -> origin/gh/H-Huang/217/base 2025-10-10T01:23:14.1154115Z * [new branch] gh/H-Huang/217/head -> origin/gh/H-Huang/217/head 2025-10-10T01:23:14.1155824Z * [new branch] gh/H-Huang/217/orig -> origin/gh/H-Huang/217/orig 2025-10-10T01:23:14.1158139Z * [new branch] gh/H-Huang/218/base -> origin/gh/H-Huang/218/base 2025-10-10T01:23:14.1160075Z * [new branch] gh/H-Huang/218/head -> origin/gh/H-Huang/218/head 2025-10-10T01:23:14.1161859Z * [new branch] gh/H-Huang/218/orig -> origin/gh/H-Huang/218/orig 2025-10-10T01:23:14.1164176Z * [new branch] gh/H-Huang/219/base -> origin/gh/H-Huang/219/base 2025-10-10T01:23:14.1165911Z * [new branch] gh/H-Huang/219/head -> origin/gh/H-Huang/219/head 2025-10-10T01:23:14.1167664Z * [new branch] gh/H-Huang/219/orig -> origin/gh/H-Huang/219/orig 2025-10-10T01:23:14.1169916Z * [new branch] gh/H-Huang/220/base -> origin/gh/H-Huang/220/base 2025-10-10T01:23:14.1171602Z * [new branch] gh/H-Huang/220/head -> origin/gh/H-Huang/220/head 2025-10-10T01:23:14.1173318Z * [new branch] gh/H-Huang/220/orig -> origin/gh/H-Huang/220/orig 2025-10-10T01:23:14.1175726Z * [new branch] gh/H-Huang/221/base -> origin/gh/H-Huang/221/base 2025-10-10T01:23:14.1177398Z * [new branch] gh/H-Huang/221/head -> origin/gh/H-Huang/221/head 2025-10-10T01:23:14.1179250Z * [new branch] gh/H-Huang/221/orig -> origin/gh/H-Huang/221/orig 2025-10-10T01:23:14.1181967Z * [new branch] gh/H-Huang/222/base -> origin/gh/H-Huang/222/base 2025-10-10T01:23:14.1183676Z * [new branch] gh/H-Huang/222/head -> origin/gh/H-Huang/222/head 2025-10-10T01:23:14.1185388Z * [new branch] gh/H-Huang/222/orig -> origin/gh/H-Huang/222/orig 2025-10-10T01:23:14.1187755Z * [new branch] gh/H-Huang/223/base -> origin/gh/H-Huang/223/base 2025-10-10T01:23:14.1189494Z * [new branch] gh/H-Huang/223/head -> origin/gh/H-Huang/223/head 2025-10-10T01:23:14.1204614Z * [new branch] gh/H-Huang/223/orig -> origin/gh/H-Huang/223/orig 2025-10-10T01:23:14.1205301Z * [new branch] gh/IvanKobzarev/115/base -> origin/gh/IvanKobzarev/115/base 2025-10-10T01:23:14.1205741Z * [new branch] gh/IvanKobzarev/115/head -> origin/gh/IvanKobzarev/115/head 2025-10-10T01:23:14.1206153Z * [new branch] gh/IvanKobzarev/115/orig -> origin/gh/IvanKobzarev/115/orig 2025-10-10T01:23:14.1206546Z * [new branch] gh/IvanKobzarev/116/base -> origin/gh/IvanKobzarev/116/base 2025-10-10T01:23:14.1206957Z * [new branch] gh/IvanKobzarev/116/head -> origin/gh/IvanKobzarev/116/head 2025-10-10T01:23:14.1207356Z * [new branch] gh/IvanKobzarev/116/orig -> origin/gh/IvanKobzarev/116/orig 2025-10-10T01:23:14.1207771Z * [new branch] gh/IvanKobzarev/118/base -> origin/gh/IvanKobzarev/118/base 2025-10-10T01:23:14.1208180Z * [new branch] gh/IvanKobzarev/118/head -> origin/gh/IvanKobzarev/118/head 2025-10-10T01:23:14.1209852Z * [new branch] gh/IvanKobzarev/118/orig -> origin/gh/IvanKobzarev/118/orig 2025-10-10T01:23:14.1212346Z * [new branch] gh/IvanKobzarev/126/base -> origin/gh/IvanKobzarev/126/base 2025-10-10T01:23:14.1214146Z * [new branch] gh/IvanKobzarev/126/head -> origin/gh/IvanKobzarev/126/head 2025-10-10T01:23:14.1215845Z * [new branch] gh/IvanKobzarev/126/orig -> origin/gh/IvanKobzarev/126/orig 2025-10-10T01:23:14.1218336Z * [new branch] gh/IvanKobzarev/127/base -> origin/gh/IvanKobzarev/127/base 2025-10-10T01:23:14.1220075Z * [new branch] gh/IvanKobzarev/127/head -> origin/gh/IvanKobzarev/127/head 2025-10-10T01:23:14.1221740Z * [new branch] gh/IvanKobzarev/127/orig -> origin/gh/IvanKobzarev/127/orig 2025-10-10T01:23:14.1224092Z * [new branch] gh/IvanKobzarev/128/base -> origin/gh/IvanKobzarev/128/base 2025-10-10T01:23:14.1225783Z * [new branch] gh/IvanKobzarev/128/head -> origin/gh/IvanKobzarev/128/head 2025-10-10T01:23:14.1227477Z * [new branch] gh/IvanKobzarev/128/orig -> origin/gh/IvanKobzarev/128/orig 2025-10-10T01:23:14.1230026Z * [new branch] gh/IvanKobzarev/135/base -> origin/gh/IvanKobzarev/135/base 2025-10-10T01:23:14.1231874Z * [new branch] gh/IvanKobzarev/135/head -> origin/gh/IvanKobzarev/135/head 2025-10-10T01:23:14.1233541Z * [new branch] gh/IvanKobzarev/135/orig -> origin/gh/IvanKobzarev/135/orig 2025-10-10T01:23:14.1236077Z * [new branch] gh/IvanKobzarev/138/base -> origin/gh/IvanKobzarev/138/base 2025-10-10T01:23:14.1237686Z * [new branch] gh/IvanKobzarev/138/head -> origin/gh/IvanKobzarev/138/head 2025-10-10T01:23:14.1239533Z * [new branch] gh/IvanKobzarev/138/orig -> origin/gh/IvanKobzarev/138/orig 2025-10-10T01:23:14.1242046Z * [new branch] gh/IvanKobzarev/141/base -> origin/gh/IvanKobzarev/141/base 2025-10-10T01:23:14.1243724Z * [new branch] gh/IvanKobzarev/141/head -> origin/gh/IvanKobzarev/141/head 2025-10-10T01:23:14.1245473Z * [new branch] gh/IvanKobzarev/141/orig -> origin/gh/IvanKobzarev/141/orig 2025-10-10T01:23:14.1248177Z * [new branch] gh/IvanKobzarev/142/base -> origin/gh/IvanKobzarev/142/base 2025-10-10T01:23:14.1249908Z * [new branch] gh/IvanKobzarev/142/head -> origin/gh/IvanKobzarev/142/head 2025-10-10T01:23:14.1251631Z * [new branch] gh/IvanKobzarev/142/orig -> origin/gh/IvanKobzarev/142/orig 2025-10-10T01:23:14.1254151Z * [new branch] gh/IvanKobzarev/144/base -> origin/gh/IvanKobzarev/144/base 2025-10-10T01:23:14.1255887Z * [new branch] gh/IvanKobzarev/144/head -> origin/gh/IvanKobzarev/144/head 2025-10-10T01:23:14.1257617Z * [new branch] gh/IvanKobzarev/144/orig -> origin/gh/IvanKobzarev/144/orig 2025-10-10T01:23:14.1260006Z * [new branch] gh/IvanKobzarev/145/base -> origin/gh/IvanKobzarev/145/base 2025-10-10T01:23:14.1261756Z * [new branch] gh/IvanKobzarev/145/head -> origin/gh/IvanKobzarev/145/head 2025-10-10T01:23:14.1263486Z * [new branch] gh/IvanKobzarev/145/orig -> origin/gh/IvanKobzarev/145/orig 2025-10-10T01:23:14.1265955Z * [new branch] gh/IvanKobzarev/146/base -> origin/gh/IvanKobzarev/146/base 2025-10-10T01:23:14.1267664Z * [new branch] gh/IvanKobzarev/146/head -> origin/gh/IvanKobzarev/146/head 2025-10-10T01:23:14.1269424Z * [new branch] gh/IvanKobzarev/146/orig -> origin/gh/IvanKobzarev/146/orig 2025-10-10T01:23:14.1271799Z * [new branch] gh/IvanKobzarev/147/base -> origin/gh/IvanKobzarev/147/base 2025-10-10T01:23:14.1273524Z * [new branch] gh/IvanKobzarev/147/head -> origin/gh/IvanKobzarev/147/head 2025-10-10T01:23:14.1275210Z * [new branch] gh/IvanKobzarev/147/orig -> origin/gh/IvanKobzarev/147/orig 2025-10-10T01:23:14.1277610Z * [new branch] gh/IvanKobzarev/148/base -> origin/gh/IvanKobzarev/148/base 2025-10-10T01:23:14.1279420Z * [new branch] gh/IvanKobzarev/148/head -> origin/gh/IvanKobzarev/148/head 2025-10-10T01:23:14.1281707Z * [new branch] gh/IvanKobzarev/149/base -> origin/gh/IvanKobzarev/149/base 2025-10-10T01:23:14.1283643Z * [new branch] gh/IvanKobzarev/149/head -> origin/gh/IvanKobzarev/149/head 2025-10-10T01:23:14.1286305Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-10-10T01:23:14.1288258Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-10-10T01:23:14.1289963Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-10-10T01:23:14.1292425Z * [new branch] gh/IvanKobzarev/151/base -> origin/gh/IvanKobzarev/151/base 2025-10-10T01:23:14.1294187Z * [new branch] gh/IvanKobzarev/151/head -> origin/gh/IvanKobzarev/151/head 2025-10-10T01:23:14.1295872Z * [new branch] gh/IvanKobzarev/151/orig -> origin/gh/IvanKobzarev/151/orig 2025-10-10T01:23:14.1298633Z * [new branch] gh/IvanKobzarev/152/base -> origin/gh/IvanKobzarev/152/base 2025-10-10T01:23:14.1300261Z * [new branch] gh/IvanKobzarev/152/head -> origin/gh/IvanKobzarev/152/head 2025-10-10T01:23:14.1301985Z * [new branch] gh/IvanKobzarev/152/orig -> origin/gh/IvanKobzarev/152/orig 2025-10-10T01:23:14.1304564Z * [new branch] gh/IvanKobzarev/153/base -> origin/gh/IvanKobzarev/153/base 2025-10-10T01:23:14.1305964Z * [new branch] gh/IvanKobzarev/153/head -> origin/gh/IvanKobzarev/153/head 2025-10-10T01:23:14.1307823Z * [new branch] gh/IvanKobzarev/153/orig -> origin/gh/IvanKobzarev/153/orig 2025-10-10T01:23:14.1310298Z * [new branch] gh/IvanKobzarev/154/base -> origin/gh/IvanKobzarev/154/base 2025-10-10T01:23:14.1312144Z * [new branch] gh/IvanKobzarev/154/head -> origin/gh/IvanKobzarev/154/head 2025-10-10T01:23:14.1313815Z * [new branch] gh/IvanKobzarev/154/orig -> origin/gh/IvanKobzarev/154/orig 2025-10-10T01:23:14.1316187Z * [new branch] gh/IvanKobzarev/155/base -> origin/gh/IvanKobzarev/155/base 2025-10-10T01:23:14.1317887Z * [new branch] gh/IvanKobzarev/155/head -> origin/gh/IvanKobzarev/155/head 2025-10-10T01:23:14.1319748Z * [new branch] gh/IvanKobzarev/155/orig -> origin/gh/IvanKobzarev/155/orig 2025-10-10T01:23:14.1322331Z * [new branch] gh/IvanKobzarev/156/base -> origin/gh/IvanKobzarev/156/base 2025-10-10T01:23:14.1324216Z * [new branch] gh/IvanKobzarev/156/head -> origin/gh/IvanKobzarev/156/head 2025-10-10T01:23:14.1325976Z * [new branch] gh/IvanKobzarev/156/orig -> origin/gh/IvanKobzarev/156/orig 2025-10-10T01:23:14.1328381Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-10-10T01:23:14.1330106Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-10-10T01:23:14.1331751Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-10-10T01:23:14.1334196Z * [new branch] gh/IvanKobzarev/158/base -> origin/gh/IvanKobzarev/158/base 2025-10-10T01:23:14.1335866Z * [new branch] gh/IvanKobzarev/158/head -> origin/gh/IvanKobzarev/158/head 2025-10-10T01:23:14.1337589Z * [new branch] gh/IvanKobzarev/158/orig -> origin/gh/IvanKobzarev/158/orig 2025-10-10T01:23:14.1340016Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-10-10T01:23:14.1341745Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-10-10T01:23:14.1343437Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-10-10T01:23:14.1345811Z * [new branch] gh/IvanKobzarev/160/base -> origin/gh/IvanKobzarev/160/base 2025-10-10T01:23:14.1347591Z * [new branch] gh/IvanKobzarev/160/head -> origin/gh/IvanKobzarev/160/head 2025-10-10T01:23:14.1349189Z * [new branch] gh/IvanKobzarev/160/orig -> origin/gh/IvanKobzarev/160/orig 2025-10-10T01:23:14.1351977Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-10-10T01:23:14.1353815Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-10-10T01:23:14.1356084Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-10-10T01:23:14.1357759Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-10-10T01:23:14.1360327Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-10-10T01:23:14.1362097Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-10-10T01:23:14.1364845Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2025-10-10T01:23:14.1366499Z * [new branch] gh/PaliC/1/head -> origin/gh/PaliC/1/head 2025-10-10T01:23:14.1368231Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2025-10-10T01:23:14.1370583Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-10-10T01:23:14.1372309Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-10-10T01:23:14.1373977Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-10-10T01:23:14.1376332Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-10-10T01:23:14.1378027Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-10-10T01:23:14.1379730Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-10-10T01:23:14.1382105Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2025-10-10T01:23:14.1383775Z * [new branch] gh/PaliC/2/head -> origin/gh/PaliC/2/head 2025-10-10T01:23:14.1385534Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2025-10-10T01:23:14.1387923Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-10-10T01:23:14.1389615Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-10-10T01:23:14.1391303Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-10-10T01:23:14.1393644Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-10-10T01:23:14.1404394Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-10-10T01:23:14.1404887Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-10-10T01:23:14.1405329Z * [new branch] gh/PaliC/22/base -> origin/gh/PaliC/22/base 2025-10-10T01:23:14.1405770Z * [new branch] gh/PaliC/22/head -> origin/gh/PaliC/22/head 2025-10-10T01:23:14.1406207Z * [new branch] gh/PaliC/22/orig -> origin/gh/PaliC/22/orig 2025-10-10T01:23:14.1406644Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-10-10T01:23:14.1408205Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-10-10T01:23:14.1409872Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-10-10T01:23:14.1412066Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-10-10T01:23:14.1413795Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-10-10T01:23:14.1415445Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-10-10T01:23:14.1417734Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-10-10T01:23:14.1419427Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-10-10T01:23:14.1421196Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-10-10T01:23:14.1423517Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-10-10T01:23:14.1425095Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-10-10T01:23:14.1426861Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-10-10T01:23:14.1429107Z * [new branch] gh/PaliC/27/head -> origin/gh/PaliC/27/head 2025-10-10T01:23:14.1430696Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-10-10T01:23:14.1432411Z * [new branch] gh/PaliC/27/orig -> origin/gh/PaliC/27/orig 2025-10-10T01:23:14.1434640Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-10-10T01:23:14.1436226Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-10-10T01:23:14.1437920Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-10-10T01:23:14.1440405Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-10-10T01:23:14.1442103Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-10-10T01:23:14.1443707Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-10-10T01:23:14.1446062Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-10-10T01:23:14.1447656Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-10-10T01:23:14.1449422Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-10-10T01:23:14.1451680Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-10-10T01:23:14.1453294Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-10-10T01:23:14.1455012Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-10-10T01:23:14.1457910Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-10-10T01:23:14.1459613Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-10-10T01:23:14.1461304Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-10-10T01:23:14.1463782Z * [new branch] gh/PaulZhang12/24/base -> origin/gh/PaulZhang12/24/base 2025-10-10T01:23:14.1465442Z * [new branch] gh/PaulZhang12/24/head -> origin/gh/PaulZhang12/24/head 2025-10-10T01:23:14.1467102Z * [new branch] gh/PaulZhang12/24/orig -> origin/gh/PaulZhang12/24/orig 2025-10-10T01:23:14.1469541Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-10-10T01:23:14.1471303Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-10-10T01:23:14.1473002Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-10-10T01:23:14.1475492Z * [new branch] gh/PaulZhang12/26/base -> origin/gh/PaulZhang12/26/base 2025-10-10T01:23:14.1477224Z * [new branch] gh/PaulZhang12/26/head -> origin/gh/PaulZhang12/26/head 2025-10-10T01:23:14.1478830Z * [new branch] gh/PaulZhang12/26/orig -> origin/gh/PaulZhang12/26/orig 2025-10-10T01:23:14.1481384Z * [new branch] gh/PaulZhang12/27/base -> origin/gh/PaulZhang12/27/base 2025-10-10T01:23:14.1483249Z * [new branch] gh/PaulZhang12/27/head -> origin/gh/PaulZhang12/27/head 2025-10-10T01:23:14.1484915Z * [new branch] gh/PaulZhang12/27/orig -> origin/gh/PaulZhang12/27/orig 2025-10-10T01:23:14.1487424Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-10-10T01:23:14.1489166Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-10-10T01:23:14.1490962Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-10-10T01:23:14.1493358Z * [new branch] gh/PaulZhang12/29/base -> origin/gh/PaulZhang12/29/base 2025-10-10T01:23:14.1494998Z * [new branch] gh/PaulZhang12/29/head -> origin/gh/PaulZhang12/29/head 2025-10-10T01:23:14.1496886Z * [new branch] gh/PaulZhang12/29/orig -> origin/gh/PaulZhang12/29/orig 2025-10-10T01:23:14.1499195Z * [new branch] gh/PaulZhang12/30/base -> origin/gh/PaulZhang12/30/base 2025-10-10T01:23:14.1500975Z * [new branch] gh/PaulZhang12/30/head -> origin/gh/PaulZhang12/30/head 2025-10-10T01:23:14.1502648Z * [new branch] gh/PaulZhang12/30/orig -> origin/gh/PaulZhang12/30/orig 2025-10-10T01:23:14.1505111Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-10-10T01:23:14.1506903Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-10-10T01:23:14.1508694Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-10-10T01:23:14.1511204Z * [new branch] gh/PaulZhang12/32/base -> origin/gh/PaulZhang12/32/base 2025-10-10T01:23:14.1512789Z * [new branch] gh/PaulZhang12/32/head -> origin/gh/PaulZhang12/32/head 2025-10-10T01:23:14.1514464Z * [new branch] gh/PaulZhang12/32/orig -> origin/gh/PaulZhang12/32/orig 2025-10-10T01:23:14.1517860Z * [new branch] gh/PaulZhang12/33/base -> origin/gh/PaulZhang12/33/base 2025-10-10T01:23:14.1520227Z * [new branch] gh/PaulZhang12/33/head -> origin/gh/PaulZhang12/33/head 2025-10-10T01:23:14.1522007Z * [new branch] gh/PaulZhang12/33/orig -> origin/gh/PaulZhang12/33/orig 2025-10-10T01:23:14.1524417Z * [new branch] gh/PaulZhang12/34/base -> origin/gh/PaulZhang12/34/base 2025-10-10T01:23:14.1526115Z * [new branch] gh/PaulZhang12/34/head -> origin/gh/PaulZhang12/34/head 2025-10-10T01:23:14.1527825Z * [new branch] gh/PaulZhang12/34/orig -> origin/gh/PaulZhang12/34/orig 2025-10-10T01:23:14.1530103Z * [new branch] gh/PaulZhang12/35/base -> origin/gh/PaulZhang12/35/base 2025-10-10T01:23:14.1531757Z * [new branch] gh/PaulZhang12/35/head -> origin/gh/PaulZhang12/35/head 2025-10-10T01:23:14.1533543Z * [new branch] gh/PaulZhang12/35/orig -> origin/gh/PaulZhang12/35/orig 2025-10-10T01:23:14.1536227Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-10-10T01:23:14.1537923Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-10-10T01:23:14.1541376Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-10-10T01:23:14.1543157Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-10-10T01:23:14.1545613Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-10-10T01:23:14.1547291Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-10-10T01:23:14.1549043Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-10-10T01:23:14.1551258Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-10-10T01:23:14.1552977Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-10-10T01:23:14.1554749Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-10-10T01:23:14.1556920Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-10-10T01:23:14.1558676Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-10-10T01:23:14.1560456Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-10-10T01:23:14.1562541Z * [new branch] gh/SherlockNoMad/13/base -> origin/gh/SherlockNoMad/13/base 2025-10-10T01:23:14.1564299Z * [new branch] gh/SherlockNoMad/13/head -> origin/gh/SherlockNoMad/13/head 2025-10-10T01:23:14.1566006Z * [new branch] gh/SherlockNoMad/13/orig -> origin/gh/SherlockNoMad/13/orig 2025-10-10T01:23:14.1568181Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-10-10T01:23:14.1569897Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-10-10T01:23:14.1572103Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-10-10T01:23:14.1573811Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-10-10T01:23:14.1576077Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-10-10T01:23:14.1577758Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-10-10T01:23:14.1580011Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-10-10T01:23:14.1581673Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-10-10T01:23:14.1583802Z * [new branch] gh/SherlockNoMad/6/base -> origin/gh/SherlockNoMad/6/base 2025-10-10T01:23:14.1585560Z * [new branch] gh/SherlockNoMad/6/head -> origin/gh/SherlockNoMad/6/head 2025-10-10T01:23:14.1587217Z * [new branch] gh/SherlockNoMad/6/orig -> origin/gh/SherlockNoMad/6/orig 2025-10-10T01:23:14.1589518Z * [new branch] gh/SherlockNoMad/7/base -> origin/gh/SherlockNoMad/7/base 2025-10-10T01:23:14.1591227Z * [new branch] gh/SherlockNoMad/7/head -> origin/gh/SherlockNoMad/7/head 2025-10-10T01:23:14.1592951Z * [new branch] gh/SherlockNoMad/7/orig -> origin/gh/SherlockNoMad/7/orig 2025-10-10T01:23:14.1595146Z * [new branch] gh/SherlockNoMad/8/base -> origin/gh/SherlockNoMad/8/base 2025-10-10T01:23:14.1596881Z * [new branch] gh/SherlockNoMad/8/head -> origin/gh/SherlockNoMad/8/head 2025-10-10T01:23:14.1598865Z * [new branch] gh/SherlockNoMad/8/orig -> origin/gh/SherlockNoMad/8/orig 2025-10-10T01:23:14.1601212Z * [new branch] gh/SherlockNoMad/9/base -> origin/gh/SherlockNoMad/9/base 2025-10-10T01:23:14.1602897Z * [new branch] gh/SherlockNoMad/9/orig -> origin/gh/SherlockNoMad/9/orig 2025-10-10T01:23:14.1605773Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-10-10T01:23:14.1607923Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-10-10T01:23:14.1610123Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-10-10T01:23:14.1612431Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-10-10T01:23:14.1615329Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-10-10T01:23:14.1617084Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-10-10T01:23:14.1619405Z * [new branch] gh/StrongerXi/133/base -> origin/gh/StrongerXi/133/base 2025-10-10T01:23:14.1621181Z * [new branch] gh/StrongerXi/133/head -> origin/gh/StrongerXi/133/head 2025-10-10T01:23:14.1622881Z * [new branch] gh/StrongerXi/133/orig -> origin/gh/StrongerXi/133/orig 2025-10-10T01:23:14.1625267Z * [new branch] gh/StrongerXi/134/base -> origin/gh/StrongerXi/134/base 2025-10-10T01:23:14.1626928Z * [new branch] gh/StrongerXi/134/head -> origin/gh/StrongerXi/134/head 2025-10-10T01:23:14.1628619Z * [new branch] gh/StrongerXi/134/orig -> origin/gh/StrongerXi/134/orig 2025-10-10T01:23:14.1630794Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-10-10T01:23:14.1632601Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-10-10T01:23:14.1634281Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-10-10T01:23:14.1636533Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-10-10T01:23:14.1638289Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-10-10T01:23:14.1640177Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-10-10T01:23:14.1642392Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-10-10T01:23:14.1644093Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-10-10T01:23:14.1646236Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-10-10T01:23:14.1648596Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-10-10T01:23:14.1650429Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-10-10T01:23:14.1652630Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-10-10T01:23:14.1654188Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-10-10T01:23:14.1656967Z * [new branch] gh/XilunWu/147/base -> origin/gh/XilunWu/147/base 2025-10-10T01:23:14.1658683Z * [new branch] gh/XilunWu/147/head -> origin/gh/XilunWu/147/head 2025-10-10T01:23:14.1660375Z * [new branch] gh/XilunWu/147/orig -> origin/gh/XilunWu/147/orig 2025-10-10T01:23:14.1662680Z * [new branch] gh/XilunWu/148/base -> origin/gh/XilunWu/148/base 2025-10-10T01:23:14.1664353Z * [new branch] gh/XilunWu/148/head -> origin/gh/XilunWu/148/head 2025-10-10T01:23:14.1666054Z * [new branch] gh/XilunWu/148/orig -> origin/gh/XilunWu/148/orig 2025-10-10T01:23:14.1668243Z * [new branch] gh/XilunWu/149/base -> origin/gh/XilunWu/149/base 2025-10-10T01:23:14.1669915Z * [new branch] gh/XilunWu/149/head -> origin/gh/XilunWu/149/head 2025-10-10T01:23:14.1671678Z * [new branch] gh/XilunWu/149/orig -> origin/gh/XilunWu/149/orig 2025-10-10T01:23:14.1673829Z * [new branch] gh/XilunWu/150/base -> origin/gh/XilunWu/150/base 2025-10-10T01:23:14.1675564Z * [new branch] gh/XilunWu/150/head -> origin/gh/XilunWu/150/head 2025-10-10T01:23:14.1677221Z * [new branch] gh/XilunWu/150/orig -> origin/gh/XilunWu/150/orig 2025-10-10T01:23:14.1679789Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-10-10T01:23:14.1681532Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-10-10T01:23:14.1683268Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-10-10T01:23:14.1685611Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-10-10T01:23:14.1687288Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-10-10T01:23:14.1688964Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-10-10T01:23:14.1691452Z * [new branch] gh/XilunWu/153/base -> origin/gh/XilunWu/153/base 2025-10-10T01:23:14.1693223Z * [new branch] gh/XilunWu/153/head -> origin/gh/XilunWu/153/head 2025-10-10T01:23:14.1694930Z * [new branch] gh/XilunWu/153/orig -> origin/gh/XilunWu/153/orig 2025-10-10T01:23:14.1697624Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-10-10T01:23:14.1699244Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-10-10T01:23:14.1700954Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-10-10T01:23:14.1703320Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-10-10T01:23:14.1704996Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-10-10T01:23:14.1706688Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-10-10T01:23:14.1709374Z * [new branch] gh/XilunWu/166/base -> origin/gh/XilunWu/166/base 2025-10-10T01:23:14.1711065Z * [new branch] gh/XilunWu/166/head -> origin/gh/XilunWu/166/head 2025-10-10T01:23:14.1712812Z * [new branch] gh/XilunWu/166/orig -> origin/gh/XilunWu/166/orig 2025-10-10T01:23:14.1715280Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-10-10T01:23:14.1716896Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-10-10T01:23:14.1718748Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-10-10T01:23:14.1721221Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-10-10T01:23:14.1722881Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-10-10T01:23:14.1724541Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-10-10T01:23:14.1726754Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-10-10T01:23:14.1728424Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-10-10T01:23:14.1730626Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-10-10T01:23:14.1734149Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-10-10T01:23:14.1735935Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-10-10T01:23:14.1737673Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-10-10T01:23:14.1739936Z * [new branch] gh/XilunWu/172/base -> origin/gh/XilunWu/172/base 2025-10-10T01:23:14.1742388Z * [new branch] gh/XilunWu/172/head -> origin/gh/XilunWu/172/head 2025-10-10T01:23:14.1744654Z * [new branch] gh/XilunWu/172/orig -> origin/gh/XilunWu/172/orig 2025-10-10T01:23:14.1746886Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-10-10T01:23:14.1748613Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-10-10T01:23:14.1750315Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-10-10T01:23:14.1753226Z * [new branch] gh/XilunWu/174/base -> origin/gh/XilunWu/174/base 2025-10-10T01:23:14.1754970Z * [new branch] gh/XilunWu/174/head -> origin/gh/XilunWu/174/head 2025-10-10T01:23:14.1756815Z * [new branch] gh/XilunWu/174/orig -> origin/gh/XilunWu/174/orig 2025-10-10T01:23:14.1759326Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-10-10T01:23:14.1761104Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-10-10T01:23:14.1762692Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-10-10T01:23:14.1765761Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-10-10T01:23:14.1767501Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-10-10T01:23:14.1769240Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-10-10T01:23:14.1771611Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-10-10T01:23:14.1773293Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-10-10T01:23:14.1775041Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-10-10T01:23:14.1777460Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-10-10T01:23:14.1779158Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-10-10T01:23:14.1780967Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-10-10T01:23:14.1783316Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-10-10T01:23:14.1785373Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-10-10T01:23:14.1786949Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-10-10T01:23:14.1789446Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-10-10T01:23:14.1791073Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-10-10T01:23:14.1792612Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-10-10T01:23:14.1795065Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-10-10T01:23:14.1796877Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-10-10T01:23:14.1800325Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-10-10T01:23:14.1802472Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-10-10T01:23:14.1804163Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-10-10T01:23:14.1805937Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-10-10T01:23:14.1808222Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-10-10T01:23:14.1809918Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-10-10T01:23:14.1811696Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-10-10T01:23:14.1814021Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-10-10T01:23:14.1815680Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-10-10T01:23:14.1817428Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-10-10T01:23:14.1819873Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-10-10T01:23:14.1821573Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-10-10T01:23:14.1823356Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-10-10T01:23:14.1825613Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-10-10T01:23:14.1827303Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-10-10T01:23:14.1829126Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-10-10T01:23:14.1831435Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-10-10T01:23:14.1833155Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-10-10T01:23:14.1835268Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-10-10T01:23:14.1837192Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-10-10T01:23:14.1838911Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-10-10T01:23:14.1840821Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-10-10T01:23:14.1842966Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-10-10T01:23:14.1844692Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-10-10T01:23:14.1846407Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-10-10T01:23:14.1848750Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-10-10T01:23:14.1850437Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-10-10T01:23:14.1852217Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-10-10T01:23:14.1854525Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-10-10T01:23:14.1856277Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-10-10T01:23:14.1857967Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-10-10T01:23:14.1860432Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-10-10T01:23:14.1862121Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-10-10T01:23:14.1863758Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-10-10T01:23:14.1866504Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-10-10T01:23:14.1868226Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-10-10T01:23:14.1869998Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-10-10T01:23:14.1872589Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-10-10T01:23:14.1874317Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-10-10T01:23:14.1876126Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-10-10T01:23:14.1878520Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-10-10T01:23:14.1880432Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-10-10T01:23:14.1882142Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-10-10T01:23:14.1884557Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-10-10T01:23:14.1886234Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-10-10T01:23:14.1888617Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-10-10T01:23:14.1890365Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-10-10T01:23:14.1892093Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-10-10T01:23:14.1894492Z * [new branch] gh/XuehaiPan/384/base -> origin/gh/XuehaiPan/384/base 2025-10-10T01:23:14.1896375Z * [new branch] gh/XuehaiPan/384/head -> origin/gh/XuehaiPan/384/head 2025-10-10T01:23:14.1898309Z * [new branch] gh/XuehaiPan/384/orig -> origin/gh/XuehaiPan/384/orig 2025-10-10T01:23:14.1900988Z * [new branch] gh/XuehaiPan/385/base -> origin/gh/XuehaiPan/385/base 2025-10-10T01:23:14.1902757Z * [new branch] gh/XuehaiPan/385/head -> origin/gh/XuehaiPan/385/head 2025-10-10T01:23:14.1904480Z * [new branch] gh/XuehaiPan/385/orig -> origin/gh/XuehaiPan/385/orig 2025-10-10T01:23:14.1906920Z * [new branch] gh/XuehaiPan/386/base -> origin/gh/XuehaiPan/386/base 2025-10-10T01:23:14.1908565Z * [new branch] gh/XuehaiPan/386/head -> origin/gh/XuehaiPan/386/head 2025-10-10T01:23:14.1910282Z * [new branch] gh/XuehaiPan/386/orig -> origin/gh/XuehaiPan/386/orig 2025-10-10T01:23:14.1912723Z * [new branch] gh/XuehaiPan/387/base -> origin/gh/XuehaiPan/387/base 2025-10-10T01:23:14.1914471Z * [new branch] gh/XuehaiPan/387/head -> origin/gh/XuehaiPan/387/head 2025-10-10T01:23:14.1916189Z * [new branch] gh/XuehaiPan/387/orig -> origin/gh/XuehaiPan/387/orig 2025-10-10T01:23:14.1918587Z * [new branch] gh/XuehaiPan/388/base -> origin/gh/XuehaiPan/388/base 2025-10-10T01:23:14.1920448Z * [new branch] gh/XuehaiPan/388/head -> origin/gh/XuehaiPan/388/head 2025-10-10T01:23:14.1922152Z * [new branch] gh/XuehaiPan/388/orig -> origin/gh/XuehaiPan/388/orig 2025-10-10T01:23:14.1924592Z * [new branch] gh/XuehaiPan/389/base -> origin/gh/XuehaiPan/389/base 2025-10-10T01:23:14.1926282Z * [new branch] gh/XuehaiPan/389/head -> origin/gh/XuehaiPan/389/head 2025-10-10T01:23:14.1928205Z * [new branch] gh/XuehaiPan/389/orig -> origin/gh/XuehaiPan/389/orig 2025-10-10T01:23:14.1930945Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-10-10T01:23:14.1932664Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-10-10T01:23:14.1934373Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-10-10T01:23:14.1936707Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-10-10T01:23:14.1938435Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-10-10T01:23:14.1940665Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-10-10T01:23:14.1942327Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-10-10T01:23:14.1944774Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-10-10T01:23:14.1946452Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-10-10T01:23:14.1948769Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-10-10T01:23:14.1950505Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-10-10T01:23:14.1952783Z * [new branch] gh/ZhiweiYan-96/64/base -> origin/gh/ZhiweiYan-96/64/base 2025-10-10T01:23:14.1954501Z * [new branch] gh/ZhiweiYan-96/64/head -> origin/gh/ZhiweiYan-96/64/head 2025-10-10T01:23:14.1956185Z * [new branch] gh/ZhiweiYan-96/64/orig -> origin/gh/ZhiweiYan-96/64/orig 2025-10-10T01:23:14.1958562Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-10-10T01:23:14.1960500Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-10-10T01:23:14.1963011Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-10-10T01:23:14.1964707Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-10-10T01:23:14.1967217Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-10-10T01:23:14.1968850Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-10-10T01:23:14.1970763Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-10-10T01:23:14.1973724Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-10-10T01:23:14.1975519Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-10-10T01:23:14.1977779Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-10-10T01:23:14.1979371Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-10-10T01:23:14.1981717Z * [new branch] gh/aakhundov/3/base -> origin/gh/aakhundov/3/base 2025-10-10T01:23:14.1983443Z * [new branch] gh/aakhundov/3/head -> origin/gh/aakhundov/3/head 2025-10-10T01:23:14.1985149Z * [new branch] gh/aakhundov/3/orig -> origin/gh/aakhundov/3/orig 2025-10-10T01:23:14.1987511Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-10-10T01:23:14.1989268Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-10-10T01:23:14.1990988Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-10-10T01:23:14.1993840Z * [new branch] gh/albanD/1/base -> origin/gh/albanD/1/base 2025-10-10T01:23:14.1995493Z * [new branch] gh/albanD/1/head -> origin/gh/albanD/1/head 2025-10-10T01:23:14.1997510Z * [new branch] gh/albanD/1/orig -> origin/gh/albanD/1/orig 2025-10-10T01:23:14.2000403Z * [new branch] gh/albanD/2/base -> origin/gh/albanD/2/base 2025-10-10T01:23:14.2001707Z * [new branch] gh/albanD/2/head -> origin/gh/albanD/2/head 2025-10-10T01:23:14.2003368Z * [new branch] gh/albanD/2/orig -> origin/gh/albanD/2/orig 2025-10-10T01:23:14.2006026Z * [new branch] gh/albanD/3/base -> origin/gh/albanD/3/base 2025-10-10T01:23:14.2007749Z * [new branch] gh/albanD/3/head -> origin/gh/albanD/3/head 2025-10-10T01:23:14.2009466Z * [new branch] gh/albanD/3/orig -> origin/gh/albanD/3/orig 2025-10-10T01:23:14.2011749Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-10-10T01:23:14.2013423Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-10-10T01:23:14.2015186Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-10-10T01:23:14.2017784Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-10-10T01:23:14.2020469Z * [new branch] gh/alexsamardzic/10/base -> origin/gh/alexsamardzic/10/base 2025-10-10T01:23:14.2022324Z * [new branch] gh/alexsamardzic/10/head -> origin/gh/alexsamardzic/10/head 2025-10-10T01:23:14.2024005Z * [new branch] gh/alexsamardzic/10/orig -> origin/gh/alexsamardzic/10/orig 2025-10-10T01:23:14.2026317Z * [new branch] gh/alexsamardzic/11/base -> origin/gh/alexsamardzic/11/base 2025-10-10T01:23:14.2028095Z * [new branch] gh/alexsamardzic/11/head -> origin/gh/alexsamardzic/11/head 2025-10-10T01:23:14.2029778Z * [new branch] gh/alexsamardzic/11/orig -> origin/gh/alexsamardzic/11/orig 2025-10-10T01:23:14.2032042Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-10-10T01:23:14.2033858Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-10-10T01:23:14.2035563Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-10-10T01:23:14.2038456Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-10-10T01:23:14.2040287Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-10-10T01:23:14.2041897Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-10-10T01:23:14.2044879Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-10-10T01:23:14.2046636Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-10-10T01:23:14.2048420Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-10-10T01:23:14.2050910Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-10-10T01:23:14.2052700Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-10-10T01:23:14.2054483Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-10-10T01:23:14.2057393Z * [new branch] gh/andyanwang/28/base -> origin/gh/andyanwang/28/base 2025-10-10T01:23:14.2059141Z * [new branch] gh/andyanwang/28/head -> origin/gh/andyanwang/28/head 2025-10-10T01:23:14.2060961Z * [new branch] gh/andyanwang/28/orig -> origin/gh/andyanwang/28/orig 2025-10-10T01:23:14.2063356Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-10-10T01:23:14.2065241Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-10-10T01:23:14.2067588Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-10-10T01:23:14.2069506Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-10-10T01:23:14.2072380Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-10-10T01:23:14.2073982Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-10-10T01:23:14.2075777Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-10-10T01:23:14.2078262Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-10-10T01:23:14.2080157Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-10-10T01:23:14.2081857Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-10-10T01:23:14.2084711Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-10-10T01:23:14.2086415Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-10-10T01:23:14.2088743Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-10-10T01:23:14.2090613Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-10-10T01:23:14.2092233Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-10-10T01:23:14.2094584Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-10-10T01:23:14.2096517Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-10-10T01:23:14.2098183Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-10-10T01:23:14.2100734Z * [new branch] gh/angelayi/117/base -> origin/gh/angelayi/117/base 2025-10-10T01:23:14.2102474Z * [new branch] gh/angelayi/117/head -> origin/gh/angelayi/117/head 2025-10-10T01:23:14.2104191Z * [new branch] gh/angelayi/117/orig -> origin/gh/angelayi/117/orig 2025-10-10T01:23:14.2106555Z * [new branch] gh/angelayi/118/base -> origin/gh/angelayi/118/base 2025-10-10T01:23:14.2108298Z * [new branch] gh/angelayi/118/head -> origin/gh/angelayi/118/head 2025-10-10T01:23:14.2110067Z * [new branch] gh/angelayi/118/orig -> origin/gh/angelayi/118/orig 2025-10-10T01:23:14.2112365Z * [new branch] gh/angelayi/119/base -> origin/gh/angelayi/119/base 2025-10-10T01:23:14.2114074Z * [new branch] gh/angelayi/119/head -> origin/gh/angelayi/119/head 2025-10-10T01:23:14.2115815Z * [new branch] gh/angelayi/119/orig -> origin/gh/angelayi/119/orig 2025-10-10T01:23:14.2118108Z * [new branch] gh/angelayi/120/base -> origin/gh/angelayi/120/base 2025-10-10T01:23:14.2119978Z * [new branch] gh/angelayi/120/head -> origin/gh/angelayi/120/head 2025-10-10T01:23:14.2121783Z * [new branch] gh/angelayi/120/orig -> origin/gh/angelayi/120/orig 2025-10-10T01:23:14.2124272Z * [new branch] gh/angelayi/121/base -> origin/gh/angelayi/121/base 2025-10-10T01:23:14.2125970Z * [new branch] gh/angelayi/121/head -> origin/gh/angelayi/121/head 2025-10-10T01:23:14.2127632Z * [new branch] gh/angelayi/121/orig -> origin/gh/angelayi/121/orig 2025-10-10T01:23:14.2129982Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-10-10T01:23:14.2131735Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-10-10T01:23:14.2133491Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-10-10T01:23:14.2135699Z * [new branch] gh/angelayi/123/base -> origin/gh/angelayi/123/base 2025-10-10T01:23:14.2137503Z * [new branch] gh/angelayi/123/head -> origin/gh/angelayi/123/head 2025-10-10T01:23:14.2139153Z * [new branch] gh/angelayi/123/orig -> origin/gh/angelayi/123/orig 2025-10-10T01:23:14.2141621Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-10-10T01:23:14.2143253Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-10-10T01:23:14.2144927Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-10-10T01:23:14.2147571Z * [new branch] gh/angelayi/125/base -> origin/gh/angelayi/125/base 2025-10-10T01:23:14.2149356Z * [new branch] gh/angelayi/125/head -> origin/gh/angelayi/125/head 2025-10-10T01:23:14.2151064Z * [new branch] gh/angelayi/125/orig -> origin/gh/angelayi/125/orig 2025-10-10T01:23:14.2153486Z * [new branch] gh/angelayi/126/base -> origin/gh/angelayi/126/base 2025-10-10T01:23:14.2155173Z * [new branch] gh/angelayi/126/head -> origin/gh/angelayi/126/head 2025-10-10T01:23:14.2156867Z * [new branch] gh/angelayi/126/orig -> origin/gh/angelayi/126/orig 2025-10-10T01:23:14.2159522Z * [new branch] gh/angelayi/127/base -> origin/gh/angelayi/127/base 2025-10-10T01:23:14.2161228Z * [new branch] gh/angelayi/127/head -> origin/gh/angelayi/127/head 2025-10-10T01:23:14.2162956Z * [new branch] gh/angelayi/127/orig -> origin/gh/angelayi/127/orig 2025-10-10T01:23:14.2165229Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-10-10T01:23:14.2166950Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-10-10T01:23:14.2168639Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-10-10T01:23:14.2170975Z * [new branch] gh/angelayi/129/base -> origin/gh/angelayi/129/base 2025-10-10T01:23:14.2172643Z * [new branch] gh/angelayi/129/head -> origin/gh/angelayi/129/head 2025-10-10T01:23:14.2174396Z * [new branch] gh/angelayi/129/orig -> origin/gh/angelayi/129/orig 2025-10-10T01:23:14.2177031Z * [new branch] gh/angelayi/130/base -> origin/gh/angelayi/130/base 2025-10-10T01:23:14.2178732Z * [new branch] gh/angelayi/130/head -> origin/gh/angelayi/130/head 2025-10-10T01:23:14.2180423Z * [new branch] gh/angelayi/130/orig -> origin/gh/angelayi/130/orig 2025-10-10T01:23:14.2183349Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-10-10T01:23:14.2185025Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-10-10T01:23:14.2186792Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-10-10T01:23:14.2189360Z * [new branch] gh/anijain2305/790/base -> origin/gh/anijain2305/790/base 2025-10-10T01:23:14.2191204Z * [new branch] gh/anijain2305/790/head -> origin/gh/anijain2305/790/head 2025-10-10T01:23:14.2192891Z * [new branch] gh/anijain2305/790/orig -> origin/gh/anijain2305/790/orig 2025-10-10T01:23:14.2196424Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-10-10T01:23:14.2201224Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-10-10T01:23:14.2203471Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-10-10T01:23:14.2206804Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-10-10T01:23:14.2209186Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-10-10T01:23:14.2211416Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-10-10T01:23:14.2214674Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-10-10T01:23:14.2217160Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-10-10T01:23:14.2219443Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-10-10T01:23:14.2222689Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-10-10T01:23:14.2225042Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-10-10T01:23:14.2227319Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-10-10T01:23:14.2230575Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-10-10T01:23:14.2232850Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-10-10T01:23:14.2235077Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-10-10T01:23:14.2238348Z * [new branch] gh/anijain2305/855/base -> origin/gh/anijain2305/855/base 2025-10-10T01:23:14.2240947Z * [new branch] gh/anijain2305/855/head -> origin/gh/anijain2305/855/head 2025-10-10T01:23:14.2243247Z * [new branch] gh/anijain2305/855/orig -> origin/gh/anijain2305/855/orig 2025-10-10T01:23:14.2247093Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-10-10T01:23:14.2249088Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-10-10T01:23:14.2250750Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-10-10T01:23:14.2253160Z * [new branch] gh/anijain2305/867/base -> origin/gh/anijain2305/867/base 2025-10-10T01:23:14.2254907Z * [new branch] gh/anijain2305/867/head -> origin/gh/anijain2305/867/head 2025-10-10T01:23:14.2256689Z * [new branch] gh/anijain2305/867/orig -> origin/gh/anijain2305/867/orig 2025-10-10T01:23:14.2259183Z * [new branch] gh/anijain2305/868/base -> origin/gh/anijain2305/868/base 2025-10-10T01:23:14.2260974Z * [new branch] gh/anijain2305/868/head -> origin/gh/anijain2305/868/head 2025-10-10T01:23:14.2262702Z * [new branch] gh/anijain2305/868/orig -> origin/gh/anijain2305/868/orig 2025-10-10T01:23:14.2265190Z * [new branch] gh/anijain2305/869/base -> origin/gh/anijain2305/869/base 2025-10-10T01:23:14.2266870Z * [new branch] gh/anijain2305/869/head -> origin/gh/anijain2305/869/head 2025-10-10T01:23:14.2268586Z * [new branch] gh/anijain2305/869/orig -> origin/gh/anijain2305/869/orig 2025-10-10T01:23:14.2270994Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-10-10T01:23:14.2272767Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-10-10T01:23:14.2274461Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-10-10T01:23:14.2276798Z * [new branch] gh/anijain2305/871/base -> origin/gh/anijain2305/871/base 2025-10-10T01:23:14.2278518Z * [new branch] gh/anijain2305/871/head -> origin/gh/anijain2305/871/head 2025-10-10T01:23:14.2280362Z * [new branch] gh/anijain2305/871/orig -> origin/gh/anijain2305/871/orig 2025-10-10T01:23:14.2282787Z * [new branch] gh/anijain2305/872/base -> origin/gh/anijain2305/872/base 2025-10-10T01:23:14.2284461Z * [new branch] gh/anijain2305/872/head -> origin/gh/anijain2305/872/head 2025-10-10T01:23:14.2286167Z * [new branch] gh/anijain2305/872/orig -> origin/gh/anijain2305/872/orig 2025-10-10T01:23:14.2288523Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-10-10T01:23:14.2290195Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-10-10T01:23:14.2291911Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-10-10T01:23:14.2294641Z * [new branch] gh/anijain2305/874/base -> origin/gh/anijain2305/874/base 2025-10-10T01:23:14.2296420Z * [new branch] gh/anijain2305/874/head -> origin/gh/anijain2305/874/head 2025-10-10T01:23:14.2298348Z * [new branch] gh/anijain2305/874/orig -> origin/gh/anijain2305/874/orig 2025-10-10T01:23:14.2300767Z * [new branch] gh/anijain2305/875/base -> origin/gh/anijain2305/875/base 2025-10-10T01:23:14.2302596Z * [new branch] gh/anijain2305/875/head -> origin/gh/anijain2305/875/head 2025-10-10T01:23:14.2304258Z * [new branch] gh/anijain2305/875/orig -> origin/gh/anijain2305/875/orig 2025-10-10T01:23:14.2306765Z * [new branch] gh/anijain2305/876/base -> origin/gh/anijain2305/876/base 2025-10-10T01:23:14.2308481Z * [new branch] gh/anijain2305/876/head -> origin/gh/anijain2305/876/head 2025-10-10T01:23:14.2310894Z * [new branch] gh/anijain2305/877/base -> origin/gh/anijain2305/877/base 2025-10-10T01:23:14.2312585Z * [new branch] gh/anijain2305/877/head -> origin/gh/anijain2305/877/head 2025-10-10T01:23:14.2314373Z * [new branch] gh/anijain2305/877/orig -> origin/gh/anijain2305/877/orig 2025-10-10T01:23:14.2316673Z * [new branch] gh/anijain2305/878/base -> origin/gh/anijain2305/878/base 2025-10-10T01:23:14.2318330Z * [new branch] gh/anijain2305/878/head -> origin/gh/anijain2305/878/head 2025-10-10T01:23:14.2320173Z * [new branch] gh/anijain2305/878/orig -> origin/gh/anijain2305/878/orig 2025-10-10T01:23:14.2322662Z * [new branch] gh/anijain2305/879/base -> origin/gh/anijain2305/879/base 2025-10-10T01:23:14.2324414Z * [new branch] gh/anijain2305/879/head -> origin/gh/anijain2305/879/head 2025-10-10T01:23:14.2326387Z * [new branch] gh/anijain2305/879/orig -> origin/gh/anijain2305/879/orig 2025-10-10T01:23:14.2328923Z * [new branch] gh/anijain2305/880/base -> origin/gh/anijain2305/880/base 2025-10-10T01:23:14.2330702Z * [new branch] gh/anijain2305/880/head -> origin/gh/anijain2305/880/head 2025-10-10T01:23:14.2332442Z * [new branch] gh/anijain2305/880/orig -> origin/gh/anijain2305/880/orig 2025-10-10T01:23:14.2334937Z * [new branch] gh/anijain2305/881/base -> origin/gh/anijain2305/881/base 2025-10-10T01:23:14.2336606Z * [new branch] gh/anijain2305/881/head -> origin/gh/anijain2305/881/head 2025-10-10T01:23:14.2338304Z * [new branch] gh/anijain2305/881/orig -> origin/gh/anijain2305/881/orig 2025-10-10T01:23:14.2340633Z * [new branch] gh/anijain2305/882/base -> origin/gh/anijain2305/882/base 2025-10-10T01:23:14.2342243Z * [new branch] gh/anijain2305/882/head -> origin/gh/anijain2305/882/head 2025-10-10T01:23:14.2343945Z * [new branch] gh/anijain2305/882/orig -> origin/gh/anijain2305/882/orig 2025-10-10T01:23:14.2346502Z * [new branch] gh/anijain2305/883/base -> origin/gh/anijain2305/883/base 2025-10-10T01:23:14.2348170Z * [new branch] gh/anijain2305/883/head -> origin/gh/anijain2305/883/head 2025-10-10T01:23:14.2349835Z * [new branch] gh/anijain2305/883/orig -> origin/gh/anijain2305/883/orig 2025-10-10T01:23:14.2352427Z * [new branch] gh/anijain2305/884/base -> origin/gh/anijain2305/884/base 2025-10-10T01:23:14.2354119Z * [new branch] gh/anijain2305/884/head -> origin/gh/anijain2305/884/head 2025-10-10T01:23:14.2355834Z * [new branch] gh/anijain2305/884/orig -> origin/gh/anijain2305/884/orig 2025-10-10T01:23:14.2358277Z * [new branch] gh/anijain2305/885/base -> origin/gh/anijain2305/885/base 2025-10-10T01:23:14.2360293Z * [new branch] gh/anijain2305/885/head -> origin/gh/anijain2305/885/head 2025-10-10T01:23:14.2361885Z * [new branch] gh/anijain2305/885/orig -> origin/gh/anijain2305/885/orig 2025-10-10T01:23:14.2364327Z * [new branch] gh/anijain2305/886/base -> origin/gh/anijain2305/886/base 2025-10-10T01:23:14.2366171Z * [new branch] gh/anijain2305/886/head -> origin/gh/anijain2305/886/head 2025-10-10T01:23:14.2367948Z * [new branch] gh/anijain2305/886/orig -> origin/gh/anijain2305/886/orig 2025-10-10T01:23:14.2370585Z * [new branch] gh/anijain2305/887/base -> origin/gh/anijain2305/887/base 2025-10-10T01:23:14.2372574Z * [new branch] gh/anijain2305/887/head -> origin/gh/anijain2305/887/head 2025-10-10T01:23:14.2374280Z * [new branch] gh/anijain2305/887/orig -> origin/gh/anijain2305/887/orig 2025-10-10T01:23:14.2376755Z * [new branch] gh/anijain2305/888/base -> origin/gh/anijain2305/888/base 2025-10-10T01:23:14.2378544Z * [new branch] gh/anijain2305/888/head -> origin/gh/anijain2305/888/head 2025-10-10T01:23:14.2380429Z * [new branch] gh/anijain2305/888/orig -> origin/gh/anijain2305/888/orig 2025-10-10T01:23:14.2382809Z * [new branch] gh/anijain2305/889/base -> origin/gh/anijain2305/889/base 2025-10-10T01:23:14.2384631Z * [new branch] gh/anijain2305/889/head -> origin/gh/anijain2305/889/head 2025-10-10T01:23:14.2386296Z * [new branch] gh/anijain2305/889/orig -> origin/gh/anijain2305/889/orig 2025-10-10T01:23:14.2388717Z * [new branch] gh/anijain2305/890/base -> origin/gh/anijain2305/890/base 2025-10-10T01:23:14.2390678Z * [new branch] gh/anijain2305/890/head -> origin/gh/anijain2305/890/head 2025-10-10T01:23:14.2392408Z * [new branch] gh/anijain2305/890/orig -> origin/gh/anijain2305/890/orig 2025-10-10T01:23:14.2394862Z * [new branch] gh/anijain2305/891/base -> origin/gh/anijain2305/891/base 2025-10-10T01:23:14.2396754Z * [new branch] gh/anijain2305/891/head -> origin/gh/anijain2305/891/head 2025-10-10T01:23:14.2398620Z * [new branch] gh/anijain2305/891/orig -> origin/gh/anijain2305/891/orig 2025-10-10T01:23:14.2401541Z * [new branch] gh/anijain2305/892/base -> origin/gh/anijain2305/892/base 2025-10-10T01:23:14.2403028Z * [new branch] gh/anijain2305/892/head -> origin/gh/anijain2305/892/head 2025-10-10T01:23:14.2404693Z * [new branch] gh/anijain2305/892/orig -> origin/gh/anijain2305/892/orig 2025-10-10T01:23:14.2407549Z * [new branch] gh/anijain2305/893/base -> origin/gh/anijain2305/893/base 2025-10-10T01:23:14.2408858Z * [new branch] gh/anijain2305/893/head -> origin/gh/anijain2305/893/head 2025-10-10T01:23:14.2410777Z * [new branch] gh/anijain2305/893/orig -> origin/gh/anijain2305/893/orig 2025-10-10T01:23:14.2413010Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-10-10T01:23:14.2414693Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-10-10T01:23:14.2416425Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-10-10T01:23:14.2418891Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-10-10T01:23:14.2420721Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-10-10T01:23:14.2422264Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-10-10T01:23:14.2425081Z * [new branch] gh/anijain2305/896/base -> origin/gh/anijain2305/896/base 2025-10-10T01:23:14.2426443Z * [new branch] gh/anijain2305/896/head -> origin/gh/anijain2305/896/head 2025-10-10T01:23:14.2428253Z * [new branch] gh/anijain2305/896/orig -> origin/gh/anijain2305/896/orig 2025-10-10T01:23:14.2431403Z * [new branch] gh/anijain2305/897/base -> origin/gh/anijain2305/897/base 2025-10-10T01:23:14.2433019Z * [new branch] gh/anijain2305/897/head -> origin/gh/anijain2305/897/head 2025-10-10T01:23:14.2434704Z * [new branch] gh/anijain2305/897/orig -> origin/gh/anijain2305/897/orig 2025-10-10T01:23:14.2437224Z * [new branch] gh/anijain2305/898/base -> origin/gh/anijain2305/898/base 2025-10-10T01:23:14.2439077Z * [new branch] gh/anijain2305/898/head -> origin/gh/anijain2305/898/head 2025-10-10T01:23:14.2441032Z * [new branch] gh/anijain2305/898/orig -> origin/gh/anijain2305/898/orig 2025-10-10T01:23:14.2443618Z * [new branch] gh/anijain2305/899/base -> origin/gh/anijain2305/899/base 2025-10-10T01:23:14.2445372Z * [new branch] gh/anijain2305/899/head -> origin/gh/anijain2305/899/head 2025-10-10T01:23:14.2447045Z * [new branch] gh/anijain2305/899/orig -> origin/gh/anijain2305/899/orig 2025-10-10T01:23:14.2449474Z * [new branch] gh/anijain2305/900/base -> origin/gh/anijain2305/900/base 2025-10-10T01:23:14.2451233Z * [new branch] gh/anijain2305/900/head -> origin/gh/anijain2305/900/head 2025-10-10T01:23:14.2452977Z * [new branch] gh/anijain2305/900/orig -> origin/gh/anijain2305/900/orig 2025-10-10T01:23:14.2455233Z * [new branch] gh/anijain2305/901/base -> origin/gh/anijain2305/901/base 2025-10-10T01:23:14.2456943Z * [new branch] gh/anijain2305/901/head -> origin/gh/anijain2305/901/head 2025-10-10T01:23:14.2458685Z * [new branch] gh/anijain2305/901/orig -> origin/gh/anijain2305/901/orig 2025-10-10T01:23:14.2460950Z * [new branch] gh/anijain2305/902/base -> origin/gh/anijain2305/902/base 2025-10-10T01:23:14.2462740Z * [new branch] gh/anijain2305/902/head -> origin/gh/anijain2305/902/head 2025-10-10T01:23:14.2464451Z * [new branch] gh/anijain2305/902/orig -> origin/gh/anijain2305/902/orig 2025-10-10T01:23:14.2466891Z * [new branch] gh/anijain2305/903/base -> origin/gh/anijain2305/903/base 2025-10-10T01:23:14.2468563Z * [new branch] gh/anijain2305/903/head -> origin/gh/anijain2305/903/head 2025-10-10T01:23:14.2470202Z * [new branch] gh/anijain2305/903/orig -> origin/gh/anijain2305/903/orig 2025-10-10T01:23:14.2472802Z * [new branch] gh/anijain2305/904/base -> origin/gh/anijain2305/904/base 2025-10-10T01:23:14.2474535Z * [new branch] gh/anijain2305/904/head -> origin/gh/anijain2305/904/head 2025-10-10T01:23:14.2476278Z * [new branch] gh/anijain2305/904/orig -> origin/gh/anijain2305/904/orig 2025-10-10T01:23:14.2479312Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-10-10T01:23:14.2481280Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-10-10T01:23:14.2482921Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-10-10T01:23:14.2485813Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-10-10T01:23:14.2487478Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-10-10T01:23:14.2489160Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-10-10T01:23:14.2492024Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-10-10T01:23:14.2493707Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-10-10T01:23:14.2496035Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-10-10T01:23:14.2497989Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-10-10T01:23:14.2500603Z * [new branch] gh/anshul-si/29/base -> origin/gh/anshul-si/29/base 2025-10-10T01:23:14.2502282Z * [new branch] gh/anshul-si/29/head -> origin/gh/anshul-si/29/head 2025-10-10T01:23:14.2503935Z * [new branch] gh/anshul-si/29/orig -> origin/gh/anshul-si/29/orig 2025-10-10T01:23:14.2506121Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-10-10T01:23:14.2507815Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-10-10T01:23:14.2510151Z * [new branch] gh/anshul-si/30/base -> origin/gh/anshul-si/30/base 2025-10-10T01:23:14.2511878Z * [new branch] gh/anshul-si/30/head -> origin/gh/anshul-si/30/head 2025-10-10T01:23:14.2513574Z * [new branch] gh/anshul-si/30/orig -> origin/gh/anshul-si/30/orig 2025-10-10T01:23:14.2515743Z * [new branch] gh/anshul-si/31/base -> origin/gh/anshul-si/31/base 2025-10-10T01:23:14.2517591Z * [new branch] gh/anshul-si/31/head -> origin/gh/anshul-si/31/head 2025-10-10T01:23:14.2519357Z * [new branch] gh/anshul-si/31/orig -> origin/gh/anshul-si/31/orig 2025-10-10T01:23:14.2521716Z * [new branch] gh/anshul-si/32/base -> origin/gh/anshul-si/32/base 2025-10-10T01:23:14.2523362Z * [new branch] gh/anshul-si/32/head -> origin/gh/anshul-si/32/head 2025-10-10T01:23:14.2525085Z * [new branch] gh/anshul-si/32/orig -> origin/gh/anshul-si/32/orig 2025-10-10T01:23:14.2527510Z * [new branch] gh/anshul-si/33/base -> origin/gh/anshul-si/33/base 2025-10-10T01:23:14.2529265Z * [new branch] gh/anshul-si/33/head -> origin/gh/anshul-si/33/head 2025-10-10T01:23:14.2530951Z * [new branch] gh/anshul-si/33/orig -> origin/gh/anshul-si/33/orig 2025-10-10T01:23:14.2533693Z * [new branch] gh/anshul-si/34/base -> origin/gh/anshul-si/34/base 2025-10-10T01:23:14.2535894Z * [new branch] gh/anshul-si/34/head -> origin/gh/anshul-si/34/head 2025-10-10T01:23:14.2537603Z * [new branch] gh/anshul-si/34/orig -> origin/gh/anshul-si/34/orig 2025-10-10T01:23:14.2539937Z * [new branch] gh/anshul-si/35/base -> origin/gh/anshul-si/35/base 2025-10-10T01:23:14.2541735Z * [new branch] gh/anshul-si/35/head -> origin/gh/anshul-si/35/head 2025-10-10T01:23:14.2543444Z * [new branch] gh/anshul-si/35/orig -> origin/gh/anshul-si/35/orig 2025-10-10T01:23:14.2545854Z * [new branch] gh/anshul-si/36/base -> origin/gh/anshul-si/36/base 2025-10-10T01:23:14.2547536Z * [new branch] gh/anshul-si/36/head -> origin/gh/anshul-si/36/head 2025-10-10T01:23:14.2549242Z * [new branch] gh/anshul-si/36/orig -> origin/gh/anshul-si/36/orig 2025-10-10T01:23:14.2551669Z * [new branch] gh/anshul-si/37/base -> origin/gh/anshul-si/37/base 2025-10-10T01:23:14.2553322Z * [new branch] gh/anshul-si/37/head -> origin/gh/anshul-si/37/head 2025-10-10T01:23:14.2555070Z * [new branch] gh/anshul-si/37/orig -> origin/gh/anshul-si/37/orig 2025-10-10T01:23:14.2557410Z * [new branch] gh/anshul-si/38/base -> origin/gh/anshul-si/38/base 2025-10-10T01:23:14.2559293Z * [new branch] gh/anshul-si/38/head -> origin/gh/anshul-si/38/head 2025-10-10T01:23:14.2561137Z * [new branch] gh/anshul-si/38/orig -> origin/gh/anshul-si/38/orig 2025-10-10T01:23:14.2563407Z * [new branch] gh/anshul-si/39/base -> origin/gh/anshul-si/39/base 2025-10-10T01:23:14.2565166Z * [new branch] gh/anshul-si/39/head -> origin/gh/anshul-si/39/head 2025-10-10T01:23:14.2566852Z * [new branch] gh/anshul-si/39/orig -> origin/gh/anshul-si/39/orig 2025-10-10T01:23:14.2569228Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-10-10T01:23:14.2570719Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-10-10T01:23:14.2573153Z * [new branch] gh/anshul-si/40/base -> origin/gh/anshul-si/40/base 2025-10-10T01:23:14.2574881Z * [new branch] gh/anshul-si/40/head -> origin/gh/anshul-si/40/head 2025-10-10T01:23:14.2576598Z * [new branch] gh/anshul-si/40/orig -> origin/gh/anshul-si/40/orig 2025-10-10T01:23:14.2578953Z * [new branch] gh/anshul-si/41/base -> origin/gh/anshul-si/41/base 2025-10-10T01:23:14.2580798Z * [new branch] gh/anshul-si/41/head -> origin/gh/anshul-si/41/head 2025-10-10T01:23:14.2582563Z * [new branch] gh/anshul-si/41/orig -> origin/gh/anshul-si/41/orig 2025-10-10T01:23:14.2584873Z * [new branch] gh/anshul-si/42/base -> origin/gh/anshul-si/42/base 2025-10-10T01:23:14.2586608Z * [new branch] gh/anshul-si/42/head -> origin/gh/anshul-si/42/head 2025-10-10T01:23:14.2588348Z * [new branch] gh/anshul-si/42/orig -> origin/gh/anshul-si/42/orig 2025-10-10T01:23:14.2590601Z * [new branch] gh/anshul-si/43/base -> origin/gh/anshul-si/43/base 2025-10-10T01:23:14.2592150Z * [new branch] gh/anshul-si/43/head -> origin/gh/anshul-si/43/head 2025-10-10T01:23:14.2593873Z * [new branch] gh/anshul-si/43/orig -> origin/gh/anshul-si/43/orig 2025-10-10T01:23:14.2596816Z * [new branch] gh/anshul-si/44/base -> origin/gh/anshul-si/44/base 2025-10-10T01:23:14.2600508Z * [new branch] gh/anshul-si/44/head -> origin/gh/anshul-si/44/head 2025-10-10T01:23:14.2602363Z * [new branch] gh/anshul-si/44/orig -> origin/gh/anshul-si/44/orig 2025-10-10T01:23:14.2604725Z * [new branch] gh/anshul-si/45/base -> origin/gh/anshul-si/45/base 2025-10-10T01:23:14.2606568Z * [new branch] gh/anshul-si/45/head -> origin/gh/anshul-si/45/head 2025-10-10T01:23:14.2608266Z * [new branch] gh/anshul-si/45/orig -> origin/gh/anshul-si/45/orig 2025-10-10T01:23:14.2610722Z * [new branch] gh/anshul-si/46/base -> origin/gh/anshul-si/46/base 2025-10-10T01:23:14.2612660Z * [new branch] gh/anshul-si/46/head -> origin/gh/anshul-si/46/head 2025-10-10T01:23:14.2614435Z * [new branch] gh/anshul-si/46/orig -> origin/gh/anshul-si/46/orig 2025-10-10T01:23:14.2616944Z * [new branch] gh/anshul-si/47/base -> origin/gh/anshul-si/47/base 2025-10-10T01:23:14.2618678Z * [new branch] gh/anshul-si/47/head -> origin/gh/anshul-si/47/head 2025-10-10T01:23:14.2620541Z * [new branch] gh/anshul-si/47/orig -> origin/gh/anshul-si/47/orig 2025-10-10T01:23:14.2622960Z * [new branch] gh/anshul-si/48/base -> origin/gh/anshul-si/48/base 2025-10-10T01:23:14.2624662Z * [new branch] gh/anshul-si/48/head -> origin/gh/anshul-si/48/head 2025-10-10T01:23:14.2626355Z * [new branch] gh/anshul-si/48/orig -> origin/gh/anshul-si/48/orig 2025-10-10T01:23:14.2628821Z * [new branch] gh/anshul-si/49/base -> origin/gh/anshul-si/49/base 2025-10-10T01:23:14.2630620Z * [new branch] gh/anshul-si/49/head -> origin/gh/anshul-si/49/head 2025-10-10T01:23:14.2632332Z * [new branch] gh/anshul-si/49/orig -> origin/gh/anshul-si/49/orig 2025-10-10T01:23:14.2634582Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-10-10T01:23:14.2636241Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-10-10T01:23:14.2638691Z * [new branch] gh/anshul-si/50/base -> origin/gh/anshul-si/50/base 2025-10-10T01:23:14.2640639Z * [new branch] gh/anshul-si/50/head -> origin/gh/anshul-si/50/head 2025-10-10T01:23:14.2642197Z * [new branch] gh/anshul-si/50/orig -> origin/gh/anshul-si/50/orig 2025-10-10T01:23:14.2644641Z * [new branch] gh/anshul-si/51/base -> origin/gh/anshul-si/51/base 2025-10-10T01:23:14.2646402Z * [new branch] gh/anshul-si/51/head -> origin/gh/anshul-si/51/head 2025-10-10T01:23:14.2648109Z * [new branch] gh/anshul-si/51/orig -> origin/gh/anshul-si/51/orig 2025-10-10T01:23:14.2650375Z * [new branch] gh/anshul-si/52/base -> origin/gh/anshul-si/52/base 2025-10-10T01:23:14.2652197Z * [new branch] gh/anshul-si/52/head -> origin/gh/anshul-si/52/head 2025-10-10T01:23:14.2653936Z * [new branch] gh/anshul-si/52/orig -> origin/gh/anshul-si/52/orig 2025-10-10T01:23:14.2656873Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-10-10T01:23:14.2658554Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-10-10T01:23:14.2660900Z * [new branch] gh/aorenste/133/base -> origin/gh/aorenste/133/base 2025-10-10T01:23:14.2662725Z * [new branch] gh/aorenste/133/head -> origin/gh/aorenste/133/head 2025-10-10T01:23:14.2664495Z * [new branch] gh/aorenste/133/orig -> origin/gh/aorenste/133/orig 2025-10-10T01:23:14.2666746Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-10-10T01:23:14.2668566Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-10-10T01:23:14.2670286Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-10-10T01:23:14.2672431Z * [new branch] gh/aorenste/135/base -> origin/gh/aorenste/135/base 2025-10-10T01:23:14.2674210Z * [new branch] gh/aorenste/135/head -> origin/gh/aorenste/135/head 2025-10-10T01:23:14.2675955Z * [new branch] gh/aorenste/135/orig -> origin/gh/aorenste/135/orig 2025-10-10T01:23:14.2678903Z * [new branch] gh/aorenste/136/base -> origin/gh/aorenste/136/base 2025-10-10T01:23:14.2680895Z * [new branch] gh/aorenste/136/head -> origin/gh/aorenste/136/head 2025-10-10T01:23:14.2682667Z * [new branch] gh/aorenste/136/orig -> origin/gh/aorenste/136/orig 2025-10-10T01:23:14.2685359Z * [new branch] gh/aorenste/137/base -> origin/gh/aorenste/137/base 2025-10-10T01:23:14.2687104Z * [new branch] gh/aorenste/137/head -> origin/gh/aorenste/137/head 2025-10-10T01:23:14.2688817Z * [new branch] gh/aorenste/137/orig -> origin/gh/aorenste/137/orig 2025-10-10T01:23:14.2691280Z * [new branch] gh/aorenste/138/base -> origin/gh/aorenste/138/base 2025-10-10T01:23:14.2692929Z * [new branch] gh/aorenste/138/head -> origin/gh/aorenste/138/head 2025-10-10T01:23:14.2694617Z * [new branch] gh/aorenste/138/orig -> origin/gh/aorenste/138/orig 2025-10-10T01:23:14.2697097Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-10-10T01:23:14.2698938Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-10-10T01:23:14.2700629Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-10-10T01:23:14.2703456Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-10-10T01:23:14.2705181Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-10-10T01:23:14.2707452Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-10-10T01:23:14.2709191Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-10-10T01:23:14.2710872Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-10-10T01:23:14.2713694Z * [new branch] gh/bdhirsh/650/base -> origin/gh/bdhirsh/650/base 2025-10-10T01:23:14.2715624Z * [new branch] gh/bdhirsh/650/head -> origin/gh/bdhirsh/650/head 2025-10-10T01:23:14.2717338Z * [new branch] gh/bdhirsh/650/orig -> origin/gh/bdhirsh/650/orig 2025-10-10T01:23:14.2719843Z * [new branch] gh/bdhirsh/665/base -> origin/gh/bdhirsh/665/base 2025-10-10T01:23:14.2721441Z * [new branch] gh/bdhirsh/665/head -> origin/gh/bdhirsh/665/head 2025-10-10T01:23:14.2723239Z * [new branch] gh/bdhirsh/665/orig -> origin/gh/bdhirsh/665/orig 2025-10-10T01:23:14.2725811Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-10-10T01:23:14.2727485Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-10-10T01:23:14.2729279Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-10-10T01:23:14.2731535Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-10-10T01:23:14.2733242Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-10-10T01:23:14.2735009Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-10-10T01:23:14.2737389Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-10-10T01:23:14.2739091Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-10-10T01:23:14.2740821Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-10-10T01:23:14.2743454Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-10-10T01:23:14.2745242Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-10-10T01:23:14.2746982Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-10-10T01:23:14.2749336Z * [new branch] gh/bdhirsh/671/base -> origin/gh/bdhirsh/671/base 2025-10-10T01:23:14.2751138Z * [new branch] gh/bdhirsh/671/head -> origin/gh/bdhirsh/671/head 2025-10-10T01:23:14.2753155Z * [new branch] gh/bdhirsh/671/orig -> origin/gh/bdhirsh/671/orig 2025-10-10T01:23:14.2755279Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-10-10T01:23:14.2757031Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-10-10T01:23:14.2758681Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-10-10T01:23:14.2761530Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-10-10T01:23:14.2763313Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-10-10T01:23:14.2765016Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-10-10T01:23:14.2767350Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-10-10T01:23:14.2769084Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-10-10T01:23:14.2770936Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-10-10T01:23:14.2773253Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-10-10T01:23:14.2775177Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-10-10T01:23:14.2776714Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-10-10T01:23:14.2779031Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-10-10T01:23:14.2780837Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-10-10T01:23:14.2782375Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-10-10T01:23:14.2784996Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-10-10T01:23:14.2786690Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-10-10T01:23:14.2788369Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-10-10T01:23:14.2790837Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-10-10T01:23:14.2792755Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-10-10T01:23:14.2794433Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-10-10T01:23:14.2796993Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-10-10T01:23:14.2798771Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-10-10T01:23:14.2800787Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-10-10T01:23:14.2803232Z * [new branch] gh/benjaminglass1/95/base -> origin/gh/benjaminglass1/95/base 2025-10-10T01:23:14.2804828Z * [new branch] gh/benjaminglass1/95/head -> origin/gh/benjaminglass1/95/head 2025-10-10T01:23:14.2806539Z * [new branch] gh/benjaminglass1/95/orig -> origin/gh/benjaminglass1/95/orig 2025-10-10T01:23:14.2808826Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-10-10T01:23:14.2810511Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-10-10T01:23:14.2812240Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-10-10T01:23:14.2815546Z * [new branch] gh/bobrenjc93/542/base -> origin/gh/bobrenjc93/542/base 2025-10-10T01:23:14.2817281Z * [new branch] gh/bobrenjc93/542/head -> origin/gh/bobrenjc93/542/head 2025-10-10T01:23:14.2818962Z * [new branch] gh/bobrenjc93/542/orig -> origin/gh/bobrenjc93/542/orig 2025-10-10T01:23:14.2821279Z * [new branch] gh/bobrenjc93/543/base -> origin/gh/bobrenjc93/543/base 2025-10-10T01:23:14.2823124Z * [new branch] gh/bobrenjc93/543/head -> origin/gh/bobrenjc93/543/head 2025-10-10T01:23:14.2824823Z * [new branch] gh/bobrenjc93/543/orig -> origin/gh/bobrenjc93/543/orig 2025-10-10T01:23:14.2827013Z * [new branch] gh/bobrenjc93/545/base -> origin/gh/bobrenjc93/545/base 2025-10-10T01:23:14.2828819Z * [new branch] gh/bobrenjc93/545/head -> origin/gh/bobrenjc93/545/head 2025-10-10T01:23:14.2830512Z * [new branch] gh/bobrenjc93/545/orig -> origin/gh/bobrenjc93/545/orig 2025-10-10T01:23:14.2833323Z * [new branch] gh/bobrenjc93/547/base -> origin/gh/bobrenjc93/547/base 2025-10-10T01:23:14.2835045Z * [new branch] gh/bobrenjc93/547/head -> origin/gh/bobrenjc93/547/head 2025-10-10T01:23:14.2836817Z * [new branch] gh/bobrenjc93/547/orig -> origin/gh/bobrenjc93/547/orig 2025-10-10T01:23:14.2839008Z * [new branch] gh/bobrenjc93/548/base -> origin/gh/bobrenjc93/548/base 2025-10-10T01:23:14.2840888Z * [new branch] gh/bobrenjc93/548/head -> origin/gh/bobrenjc93/548/head 2025-10-10T01:23:14.2842547Z * [new branch] gh/bobrenjc93/548/orig -> origin/gh/bobrenjc93/548/orig 2025-10-10T01:23:14.2844806Z * [new branch] gh/bobrenjc93/553/base -> origin/gh/bobrenjc93/553/base 2025-10-10T01:23:14.2846480Z * [new branch] gh/bobrenjc93/553/head -> origin/gh/bobrenjc93/553/head 2025-10-10T01:23:14.2848359Z * [new branch] gh/bobrenjc93/553/orig -> origin/gh/bobrenjc93/553/orig 2025-10-10T01:23:14.2850450Z * [new branch] gh/bobrenjc93/554/base -> origin/gh/bobrenjc93/554/base 2025-10-10T01:23:14.2852037Z * [new branch] gh/bobrenjc93/554/head -> origin/gh/bobrenjc93/554/head 2025-10-10T01:23:14.2853729Z * [new branch] gh/bobrenjc93/554/orig -> origin/gh/bobrenjc93/554/orig 2025-10-10T01:23:14.2856101Z * [new branch] gh/bobrenjc93/555/base -> origin/gh/bobrenjc93/555/base 2025-10-10T01:23:14.2857747Z * [new branch] gh/bobrenjc93/555/head -> origin/gh/bobrenjc93/555/head 2025-10-10T01:23:14.2859437Z * [new branch] gh/bobrenjc93/555/orig -> origin/gh/bobrenjc93/555/orig 2025-10-10T01:23:14.2861807Z * [new branch] gh/bobrenjc93/557/base -> origin/gh/bobrenjc93/557/base 2025-10-10T01:23:14.2863523Z * [new branch] gh/bobrenjc93/557/head -> origin/gh/bobrenjc93/557/head 2025-10-10T01:23:14.2865173Z * [new branch] gh/bobrenjc93/557/orig -> origin/gh/bobrenjc93/557/orig 2025-10-10T01:23:14.2867671Z * [new branch] gh/bobrenjc93/558/base -> origin/gh/bobrenjc93/558/base 2025-10-10T01:23:14.2869392Z * [new branch] gh/bobrenjc93/558/head -> origin/gh/bobrenjc93/558/head 2025-10-10T01:23:14.2871292Z * [new branch] gh/bobrenjc93/558/orig -> origin/gh/bobrenjc93/558/orig 2025-10-10T01:23:14.2873869Z * [new branch] gh/bobrenjc93/559/base -> origin/gh/bobrenjc93/559/base 2025-10-10T01:23:14.2875540Z * [new branch] gh/bobrenjc93/559/head -> origin/gh/bobrenjc93/559/head 2025-10-10T01:23:14.2877239Z * [new branch] gh/bobrenjc93/559/orig -> origin/gh/bobrenjc93/559/orig 2025-10-10T01:23:14.2879543Z * [new branch] gh/bobrenjc93/560/base -> origin/gh/bobrenjc93/560/base 2025-10-10T01:23:14.2881546Z * [new branch] gh/bobrenjc93/560/head -> origin/gh/bobrenjc93/560/head 2025-10-10T01:23:14.2883388Z * [new branch] gh/bobrenjc93/560/orig -> origin/gh/bobrenjc93/560/orig 2025-10-10T01:23:14.2885846Z * [new branch] gh/bobrenjc93/561/base -> origin/gh/bobrenjc93/561/base 2025-10-10T01:23:14.2887550Z * [new branch] gh/bobrenjc93/561/head -> origin/gh/bobrenjc93/561/head 2025-10-10T01:23:14.2889206Z * [new branch] gh/bobrenjc93/561/orig -> origin/gh/bobrenjc93/561/orig 2025-10-10T01:23:14.2891559Z * [new branch] gh/bobrenjc93/562/base -> origin/gh/bobrenjc93/562/base 2025-10-10T01:23:14.2893383Z * [new branch] gh/bobrenjc93/562/head -> origin/gh/bobrenjc93/562/head 2025-10-10T01:23:14.2895068Z * [new branch] gh/bobrenjc93/562/orig -> origin/gh/bobrenjc93/562/orig 2025-10-10T01:23:14.2899292Z * [new branch] gh/bobrenjc93/563/base -> origin/gh/bobrenjc93/563/base 2025-10-10T01:23:14.2900911Z * [new branch] gh/bobrenjc93/563/head -> origin/gh/bobrenjc93/563/head 2025-10-10T01:23:14.2902690Z * [new branch] gh/bobrenjc93/563/orig -> origin/gh/bobrenjc93/563/orig 2025-10-10T01:23:14.2905164Z * [new branch] gh/bobrenjc93/564/base -> origin/gh/bobrenjc93/564/base 2025-10-10T01:23:14.2906832Z * [new branch] gh/bobrenjc93/564/head -> origin/gh/bobrenjc93/564/head 2025-10-10T01:23:14.2908516Z * [new branch] gh/bobrenjc93/564/orig -> origin/gh/bobrenjc93/564/orig 2025-10-10T01:23:14.2911120Z * [new branch] gh/bobrenjc93/565/base -> origin/gh/bobrenjc93/565/base 2025-10-10T01:23:14.2912890Z * [new branch] gh/bobrenjc93/565/head -> origin/gh/bobrenjc93/565/head 2025-10-10T01:23:14.2914540Z * [new branch] gh/bobrenjc93/565/orig -> origin/gh/bobrenjc93/565/orig 2025-10-10T01:23:14.2917059Z * [new branch] gh/bobrenjc93/566/base -> origin/gh/bobrenjc93/566/base 2025-10-10T01:23:14.2918643Z * [new branch] gh/bobrenjc93/566/head -> origin/gh/bobrenjc93/566/head 2025-10-10T01:23:14.2920433Z * [new branch] gh/bobrenjc93/566/orig -> origin/gh/bobrenjc93/566/orig 2025-10-10T01:23:14.2922710Z * [new branch] gh/bobrenjc93/567/base -> origin/gh/bobrenjc93/567/base 2025-10-10T01:23:14.2924408Z * [new branch] gh/bobrenjc93/567/head -> origin/gh/bobrenjc93/567/head 2025-10-10T01:23:14.2926126Z * [new branch] gh/bobrenjc93/567/orig -> origin/gh/bobrenjc93/567/orig 2025-10-10T01:23:14.2928366Z * [new branch] gh/bobrenjc93/568/base -> origin/gh/bobrenjc93/568/base 2025-10-10T01:23:14.2930510Z * [new branch] gh/bobrenjc93/568/head -> origin/gh/bobrenjc93/568/head 2025-10-10T01:23:14.2932275Z * [new branch] gh/bobrenjc93/568/orig -> origin/gh/bobrenjc93/568/orig 2025-10-10T01:23:14.2934519Z * [new branch] gh/bobrenjc93/569/base -> origin/gh/bobrenjc93/569/base 2025-10-10T01:23:14.2936218Z * [new branch] gh/bobrenjc93/569/head -> origin/gh/bobrenjc93/569/head 2025-10-10T01:23:14.2937930Z * [new branch] gh/bobrenjc93/569/orig -> origin/gh/bobrenjc93/569/orig 2025-10-10T01:23:14.2940391Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-10-10T01:23:14.2942116Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-10-10T01:23:14.2943989Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-10-10T01:23:14.2946269Z * [new branch] gh/bobrenjc93/571/base -> origin/gh/bobrenjc93/571/base 2025-10-10T01:23:14.2947978Z * [new branch] gh/bobrenjc93/571/head -> origin/gh/bobrenjc93/571/head 2025-10-10T01:23:14.2949663Z * [new branch] gh/bobrenjc93/571/orig -> origin/gh/bobrenjc93/571/orig 2025-10-10T01:23:14.2952040Z * [new branch] gh/bobrenjc93/572/base -> origin/gh/bobrenjc93/572/base 2025-10-10T01:23:14.2953802Z * [new branch] gh/bobrenjc93/572/head -> origin/gh/bobrenjc93/572/head 2025-10-10T01:23:14.2955527Z * [new branch] gh/bobrenjc93/572/orig -> origin/gh/bobrenjc93/572/orig 2025-10-10T01:23:14.2957949Z * [new branch] gh/bobrenjc93/573/base -> origin/gh/bobrenjc93/573/base 2025-10-10T01:23:14.2959799Z * [new branch] gh/bobrenjc93/573/head -> origin/gh/bobrenjc93/573/head 2025-10-10T01:23:14.2961528Z * [new branch] gh/bobrenjc93/573/orig -> origin/gh/bobrenjc93/573/orig 2025-10-10T01:23:14.2963992Z * [new branch] gh/bobrenjc93/574/base -> origin/gh/bobrenjc93/574/base 2025-10-10T01:23:14.2965689Z * [new branch] gh/bobrenjc93/574/head -> origin/gh/bobrenjc93/574/head 2025-10-10T01:23:14.2967389Z * [new branch] gh/bobrenjc93/574/orig -> origin/gh/bobrenjc93/574/orig 2025-10-10T01:23:14.2969758Z * [new branch] gh/bobrenjc93/575/base -> origin/gh/bobrenjc93/575/base 2025-10-10T01:23:14.2971498Z * [new branch] gh/bobrenjc93/575/head -> origin/gh/bobrenjc93/575/head 2025-10-10T01:23:14.2973273Z * [new branch] gh/bobrenjc93/575/orig -> origin/gh/bobrenjc93/575/orig 2025-10-10T01:23:14.2976082Z * [new branch] gh/bobrenjc93/576/base -> origin/gh/bobrenjc93/576/base 2025-10-10T01:23:14.2977780Z * [new branch] gh/bobrenjc93/576/head -> origin/gh/bobrenjc93/576/head 2025-10-10T01:23:14.2979452Z * [new branch] gh/bobrenjc93/576/orig -> origin/gh/bobrenjc93/576/orig 2025-10-10T01:23:14.2981755Z * [new branch] gh/bobrenjc93/577/base -> origin/gh/bobrenjc93/577/base 2025-10-10T01:23:14.2983566Z * [new branch] gh/bobrenjc93/577/head -> origin/gh/bobrenjc93/577/head 2025-10-10T01:23:14.2985234Z * [new branch] gh/bobrenjc93/577/orig -> origin/gh/bobrenjc93/577/orig 2025-10-10T01:23:14.3014298Z * [new branch] gh/bobrenjc93/578/base -> origin/gh/bobrenjc93/578/base 2025-10-10T01:23:14.3014708Z * [new branch] gh/bobrenjc93/578/head -> origin/gh/bobrenjc93/578/head 2025-10-10T01:23:14.3014949Z * [new branch] gh/bobrenjc93/578/orig -> origin/gh/bobrenjc93/578/orig 2025-10-10T01:23:14.3015153Z * [new branch] gh/bobrenjc93/579/base -> origin/gh/bobrenjc93/579/base 2025-10-10T01:23:14.3015346Z * [new branch] gh/bobrenjc93/579/head -> origin/gh/bobrenjc93/579/head 2025-10-10T01:23:14.3015546Z * [new branch] gh/bobrenjc93/579/orig -> origin/gh/bobrenjc93/579/orig 2025-10-10T01:23:14.3015739Z * [new branch] gh/bobrenjc93/580/base -> origin/gh/bobrenjc93/580/base 2025-10-10T01:23:14.3015990Z * [new branch] gh/bobrenjc93/580/head -> origin/gh/bobrenjc93/580/head 2025-10-10T01:23:14.3016220Z * [new branch] gh/bobrenjc93/580/orig -> origin/gh/bobrenjc93/580/orig 2025-10-10T01:23:14.3016600Z * [new branch] gh/bobrenjc93/581/base -> origin/gh/bobrenjc93/581/base 2025-10-10T01:23:14.3016803Z * [new branch] gh/bobrenjc93/581/head -> origin/gh/bobrenjc93/581/head 2025-10-10T01:23:14.3016993Z * [new branch] gh/bobrenjc93/581/orig -> origin/gh/bobrenjc93/581/orig 2025-10-10T01:23:14.3017186Z * [new branch] gh/bobrenjc93/582/base -> origin/gh/bobrenjc93/582/base 2025-10-10T01:23:14.3017372Z * [new branch] gh/bobrenjc93/582/head -> origin/gh/bobrenjc93/582/head 2025-10-10T01:23:14.3017554Z * [new branch] gh/bobrenjc93/582/orig -> origin/gh/bobrenjc93/582/orig 2025-10-10T01:23:14.3017762Z * [new branch] gh/bobrenjc93/583/base -> origin/gh/bobrenjc93/583/base 2025-10-10T01:23:14.3019106Z * [new branch] gh/bobrenjc93/583/head -> origin/gh/bobrenjc93/583/head 2025-10-10T01:23:14.3020763Z * [new branch] gh/bobrenjc93/583/orig -> origin/gh/bobrenjc93/583/orig 2025-10-10T01:23:14.3023153Z * [new branch] gh/bobrenjc93/584/base -> origin/gh/bobrenjc93/584/base 2025-10-10T01:23:14.3025046Z * [new branch] gh/bobrenjc93/584/head -> origin/gh/bobrenjc93/584/head 2025-10-10T01:23:14.3026652Z * [new branch] gh/bobrenjc93/584/orig -> origin/gh/bobrenjc93/584/orig 2025-10-10T01:23:14.3028910Z * [new branch] gh/bobrenjc93/585/base -> origin/gh/bobrenjc93/585/base 2025-10-10T01:23:14.3030595Z * [new branch] gh/bobrenjc93/585/head -> origin/gh/bobrenjc93/585/head 2025-10-10T01:23:14.3032931Z * [new branch] gh/bobrenjc93/585/orig -> origin/gh/bobrenjc93/585/orig 2025-10-10T01:23:14.3036327Z * [new branch] gh/bobrenjc93/586/base -> origin/gh/bobrenjc93/586/base 2025-10-10T01:23:14.3037935Z * [new branch] gh/bobrenjc93/586/head -> origin/gh/bobrenjc93/586/head 2025-10-10T01:23:14.3039681Z * [new branch] gh/bobrenjc93/586/orig -> origin/gh/bobrenjc93/586/orig 2025-10-10T01:23:14.3042174Z * [new branch] gh/bobrenjc93/587/base -> origin/gh/bobrenjc93/587/base 2025-10-10T01:23:14.3043883Z * [new branch] gh/bobrenjc93/587/head -> origin/gh/bobrenjc93/587/head 2025-10-10T01:23:14.3045600Z * [new branch] gh/bobrenjc93/587/orig -> origin/gh/bobrenjc93/587/orig 2025-10-10T01:23:14.3047989Z * [new branch] gh/bobrenjc93/588/base -> origin/gh/bobrenjc93/588/base 2025-10-10T01:23:14.3049733Z * [new branch] gh/bobrenjc93/588/head -> origin/gh/bobrenjc93/588/head 2025-10-10T01:23:14.3051410Z * [new branch] gh/bobrenjc93/588/orig -> origin/gh/bobrenjc93/588/orig 2025-10-10T01:23:14.3053958Z * [new branch] gh/bobrenjc93/589/base -> origin/gh/bobrenjc93/589/base 2025-10-10T01:23:14.3055579Z * [new branch] gh/bobrenjc93/589/head -> origin/gh/bobrenjc93/589/head 2025-10-10T01:23:14.3057245Z * [new branch] gh/bobrenjc93/589/orig -> origin/gh/bobrenjc93/589/orig 2025-10-10T01:23:14.3059713Z * [new branch] gh/bobrenjc93/590/base -> origin/gh/bobrenjc93/590/base 2025-10-10T01:23:14.3061418Z * [new branch] gh/bobrenjc93/590/head -> origin/gh/bobrenjc93/590/head 2025-10-10T01:23:14.3063226Z * [new branch] gh/bobrenjc93/590/orig -> origin/gh/bobrenjc93/590/orig 2025-10-10T01:23:14.3065615Z * [new branch] gh/bobrenjc93/591/base -> origin/gh/bobrenjc93/591/base 2025-10-10T01:23:14.3067285Z * [new branch] gh/bobrenjc93/591/head -> origin/gh/bobrenjc93/591/head 2025-10-10T01:23:14.3069026Z * [new branch] gh/bobrenjc93/591/orig -> origin/gh/bobrenjc93/591/orig 2025-10-10T01:23:14.3071456Z * [new branch] gh/bobrenjc93/592/base -> origin/gh/bobrenjc93/592/base 2025-10-10T01:23:14.3073179Z * [new branch] gh/bobrenjc93/592/head -> origin/gh/bobrenjc93/592/head 2025-10-10T01:23:14.3075008Z * [new branch] gh/bobrenjc93/592/orig -> origin/gh/bobrenjc93/592/orig 2025-10-10T01:23:14.3077314Z * [new branch] gh/bobrenjc93/593/base -> origin/gh/bobrenjc93/593/base 2025-10-10T01:23:14.3079049Z * [new branch] gh/bobrenjc93/593/head -> origin/gh/bobrenjc93/593/head 2025-10-10T01:23:14.3080977Z * [new branch] gh/bobrenjc93/593/orig -> origin/gh/bobrenjc93/593/orig 2025-10-10T01:23:14.3083293Z * [new branch] gh/bobrenjc93/594/base -> origin/gh/bobrenjc93/594/base 2025-10-10T01:23:14.3084994Z * [new branch] gh/bobrenjc93/594/head -> origin/gh/bobrenjc93/594/head 2025-10-10T01:23:14.3086719Z * [new branch] gh/bobrenjc93/594/orig -> origin/gh/bobrenjc93/594/orig 2025-10-10T01:23:14.3088958Z * [new branch] gh/bobrenjc93/595/base -> origin/gh/bobrenjc93/595/base 2025-10-10T01:23:14.3090600Z * [new branch] gh/bobrenjc93/595/head -> origin/gh/bobrenjc93/595/head 2025-10-10T01:23:14.3092216Z * [new branch] gh/bobrenjc93/595/orig -> origin/gh/bobrenjc93/595/orig 2025-10-10T01:23:14.3094473Z * [new branch] gh/bobrenjc93/596/base -> origin/gh/bobrenjc93/596/base 2025-10-10T01:23:14.3096376Z * [new branch] gh/bobrenjc93/596/head -> origin/gh/bobrenjc93/596/head 2025-10-10T01:23:14.3098079Z * [new branch] gh/bobrenjc93/596/orig -> origin/gh/bobrenjc93/596/orig 2025-10-10T01:23:14.3100408Z * [new branch] gh/bobrenjc93/597/base -> origin/gh/bobrenjc93/597/base 2025-10-10T01:23:14.3102184Z * [new branch] gh/bobrenjc93/597/head -> origin/gh/bobrenjc93/597/head 2025-10-10T01:23:14.3103971Z * [new branch] gh/bobrenjc93/597/orig -> origin/gh/bobrenjc93/597/orig 2025-10-10T01:23:14.3106308Z * [new branch] gh/bobrenjc93/598/base -> origin/gh/bobrenjc93/598/base 2025-10-10T01:23:14.3108037Z * [new branch] gh/bobrenjc93/598/head -> origin/gh/bobrenjc93/598/head 2025-10-10T01:23:14.3109686Z * [new branch] gh/bobrenjc93/598/orig -> origin/gh/bobrenjc93/598/orig 2025-10-10T01:23:14.3112123Z * [new branch] gh/bobrenjc93/599/base -> origin/gh/bobrenjc93/599/base 2025-10-10T01:23:14.3113838Z * [new branch] gh/bobrenjc93/599/head -> origin/gh/bobrenjc93/599/head 2025-10-10T01:23:14.3115527Z * [new branch] gh/bobrenjc93/599/orig -> origin/gh/bobrenjc93/599/orig 2025-10-10T01:23:14.3117791Z * [new branch] gh/bobrenjc93/600/base -> origin/gh/bobrenjc93/600/base 2025-10-10T01:23:14.3120350Z * [new branch] gh/bobrenjc93/600/head -> origin/gh/bobrenjc93/600/head 2025-10-10T01:23:14.3122014Z * [new branch] gh/bobrenjc93/600/orig -> origin/gh/bobrenjc93/600/orig 2025-10-10T01:23:14.3125176Z * [new branch] gh/bobrenjc93/601/base -> origin/gh/bobrenjc93/601/base 2025-10-10T01:23:14.3127403Z * [new branch] gh/bobrenjc93/601/head -> origin/gh/bobrenjc93/601/head 2025-10-10T01:23:14.3129775Z * [new branch] gh/bobrenjc93/601/orig -> origin/gh/bobrenjc93/601/orig 2025-10-10T01:23:14.3132585Z * [new branch] gh/bobrenjc93/602/base -> origin/gh/bobrenjc93/602/base 2025-10-10T01:23:14.3134340Z * [new branch] gh/bobrenjc93/602/head -> origin/gh/bobrenjc93/602/head 2025-10-10T01:23:14.3136295Z * [new branch] gh/bobrenjc93/602/orig -> origin/gh/bobrenjc93/602/orig 2025-10-10T01:23:14.3138561Z * [new branch] gh/bobrenjc93/603/base -> origin/gh/bobrenjc93/603/base 2025-10-10T01:23:14.3140263Z * [new branch] gh/bobrenjc93/603/head -> origin/gh/bobrenjc93/603/head 2025-10-10T01:23:14.3141889Z * [new branch] gh/bobrenjc93/603/orig -> origin/gh/bobrenjc93/603/orig 2025-10-10T01:23:14.3144823Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-10-10T01:23:14.3146681Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-10-10T01:23:14.3148371Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-10-10T01:23:14.3150709Z * [new branch] gh/bobrenjc93/605/base -> origin/gh/bobrenjc93/605/base 2025-10-10T01:23:14.3152510Z * [new branch] gh/bobrenjc93/605/head -> origin/gh/bobrenjc93/605/head 2025-10-10T01:23:14.3154194Z * [new branch] gh/bobrenjc93/605/orig -> origin/gh/bobrenjc93/605/orig 2025-10-10T01:23:14.3156790Z * [new branch] gh/bobrenjc93/606/base -> origin/gh/bobrenjc93/606/base 2025-10-10T01:23:14.3158430Z * [new branch] gh/bobrenjc93/606/head -> origin/gh/bobrenjc93/606/head 2025-10-10T01:23:14.3160263Z * [new branch] gh/bobrenjc93/606/orig -> origin/gh/bobrenjc93/606/orig 2025-10-10T01:23:14.3162726Z * [new branch] gh/bobrenjc93/607/base -> origin/gh/bobrenjc93/607/base 2025-10-10T01:23:14.3164474Z * [new branch] gh/bobrenjc93/607/head -> origin/gh/bobrenjc93/607/head 2025-10-10T01:23:14.3166073Z * [new branch] gh/bobrenjc93/607/orig -> origin/gh/bobrenjc93/607/orig 2025-10-10T01:23:14.3168524Z * [new branch] gh/bobrenjc93/608/base -> origin/gh/bobrenjc93/608/base 2025-10-10T01:23:14.3170541Z * [new branch] gh/bobrenjc93/608/head -> origin/gh/bobrenjc93/608/head 2025-10-10T01:23:14.3172210Z * [new branch] gh/bobrenjc93/608/orig -> origin/gh/bobrenjc93/608/orig 2025-10-10T01:23:14.3174613Z * [new branch] gh/bobrenjc93/609/base -> origin/gh/bobrenjc93/609/base 2025-10-10T01:23:14.3176315Z * [new branch] gh/bobrenjc93/609/head -> origin/gh/bobrenjc93/609/head 2025-10-10T01:23:14.3178057Z * [new branch] gh/bobrenjc93/609/orig -> origin/gh/bobrenjc93/609/orig 2025-10-10T01:23:14.3180716Z * [new branch] gh/bobrenjc93/610/base -> origin/gh/bobrenjc93/610/base 2025-10-10T01:23:14.3182380Z * [new branch] gh/bobrenjc93/610/head -> origin/gh/bobrenjc93/610/head 2025-10-10T01:23:14.3184187Z * [new branch] gh/bobrenjc93/610/orig -> origin/gh/bobrenjc93/610/orig 2025-10-10T01:23:14.3186598Z * [new branch] gh/bobrenjc93/611/base -> origin/gh/bobrenjc93/611/base 2025-10-10T01:23:14.3188231Z * [new branch] gh/bobrenjc93/611/head -> origin/gh/bobrenjc93/611/head 2025-10-10T01:23:14.3189861Z * [new branch] gh/bobrenjc93/611/orig -> origin/gh/bobrenjc93/611/orig 2025-10-10T01:23:14.3192371Z * [new branch] gh/bobrenjc93/612/base -> origin/gh/bobrenjc93/612/base 2025-10-10T01:23:14.3194079Z * [new branch] gh/bobrenjc93/612/head -> origin/gh/bobrenjc93/612/head 2025-10-10T01:23:14.3195724Z * [new branch] gh/bobrenjc93/612/orig -> origin/gh/bobrenjc93/612/orig 2025-10-10T01:23:14.3198567Z * [new branch] gh/bobrenjc93/613/base -> origin/gh/bobrenjc93/613/base 2025-10-10T01:23:14.3200359Z * [new branch] gh/bobrenjc93/613/head -> origin/gh/bobrenjc93/613/head 2025-10-10T01:23:14.3202055Z * [new branch] gh/bobrenjc93/613/orig -> origin/gh/bobrenjc93/613/orig 2025-10-10T01:23:14.3204448Z * [new branch] gh/bobrenjc93/614/base -> origin/gh/bobrenjc93/614/base 2025-10-10T01:23:14.3206513Z * [new branch] gh/bobrenjc93/614/head -> origin/gh/bobrenjc93/614/head 2025-10-10T01:23:14.3207972Z * [new branch] gh/bobrenjc93/614/orig -> origin/gh/bobrenjc93/614/orig 2025-10-10T01:23:14.3210328Z * [new branch] gh/bobrenjc93/615/base -> origin/gh/bobrenjc93/615/base 2025-10-10T01:23:14.3212076Z * [new branch] gh/bobrenjc93/615/head -> origin/gh/bobrenjc93/615/head 2025-10-10T01:23:14.3213734Z * [new branch] gh/bobrenjc93/615/orig -> origin/gh/bobrenjc93/615/orig 2025-10-10T01:23:14.3216206Z * [new branch] gh/bobrenjc93/616/base -> origin/gh/bobrenjc93/616/base 2025-10-10T01:23:14.3217945Z * [new branch] gh/bobrenjc93/616/head -> origin/gh/bobrenjc93/616/head 2025-10-10T01:23:14.3219668Z * [new branch] gh/bobrenjc93/616/orig -> origin/gh/bobrenjc93/616/orig 2025-10-10T01:23:14.3222002Z * [new branch] gh/bobrenjc93/617/base -> origin/gh/bobrenjc93/617/base 2025-10-10T01:23:14.3223714Z * [new branch] gh/bobrenjc93/617/head -> origin/gh/bobrenjc93/617/head 2025-10-10T01:23:14.3225423Z * [new branch] gh/bobrenjc93/617/orig -> origin/gh/bobrenjc93/617/orig 2025-10-10T01:23:14.3227822Z * [new branch] gh/bobrenjc93/618/base -> origin/gh/bobrenjc93/618/base 2025-10-10T01:23:14.3229526Z * [new branch] gh/bobrenjc93/618/head -> origin/gh/bobrenjc93/618/head 2025-10-10T01:23:14.3231185Z * [new branch] gh/bobrenjc93/618/orig -> origin/gh/bobrenjc93/618/orig 2025-10-10T01:23:14.3233532Z * [new branch] gh/bobrenjc93/619/base -> origin/gh/bobrenjc93/619/base 2025-10-10T01:23:14.3235248Z * [new branch] gh/bobrenjc93/619/head -> origin/gh/bobrenjc93/619/head 2025-10-10T01:23:14.3236930Z * [new branch] gh/bobrenjc93/619/orig -> origin/gh/bobrenjc93/619/orig 2025-10-10T01:23:14.3239328Z * [new branch] gh/bobrenjc93/620/base -> origin/gh/bobrenjc93/620/base 2025-10-10T01:23:14.3241108Z * [new branch] gh/bobrenjc93/620/head -> origin/gh/bobrenjc93/620/head 2025-10-10T01:23:14.3242832Z * [new branch] gh/bobrenjc93/620/orig -> origin/gh/bobrenjc93/620/orig 2025-10-10T01:23:14.3245156Z * [new branch] gh/bobrenjc93/621/base -> origin/gh/bobrenjc93/621/base 2025-10-10T01:23:14.3246903Z * [new branch] gh/bobrenjc93/621/head -> origin/gh/bobrenjc93/621/head 2025-10-10T01:23:14.3248574Z * [new branch] gh/bobrenjc93/621/orig -> origin/gh/bobrenjc93/621/orig 2025-10-10T01:23:14.3251057Z * [new branch] gh/bobrenjc93/622/base -> origin/gh/bobrenjc93/622/base 2025-10-10T01:23:14.3252811Z * [new branch] gh/bobrenjc93/622/head -> origin/gh/bobrenjc93/622/head 2025-10-10T01:23:14.3254429Z * [new branch] gh/bobrenjc93/622/orig -> origin/gh/bobrenjc93/622/orig 2025-10-10T01:23:14.3257101Z * [new branch] gh/bobrenjc93/623/base -> origin/gh/bobrenjc93/623/base 2025-10-10T01:23:14.3258933Z * [new branch] gh/bobrenjc93/623/head -> origin/gh/bobrenjc93/623/head 2025-10-10T01:23:14.3260468Z * [new branch] gh/bobrenjc93/623/orig -> origin/gh/bobrenjc93/623/orig 2025-10-10T01:23:14.3262639Z * [new branch] gh/bobrenjc93/624/base -> origin/gh/bobrenjc93/624/base 2025-10-10T01:23:14.3264393Z * [new branch] gh/bobrenjc93/624/head -> origin/gh/bobrenjc93/624/head 2025-10-10T01:23:14.3266200Z * [new branch] gh/bobrenjc93/624/orig -> origin/gh/bobrenjc93/624/orig 2025-10-10T01:23:14.3268595Z * [new branch] gh/bobrenjc93/625/base -> origin/gh/bobrenjc93/625/base 2025-10-10T01:23:14.3270206Z * [new branch] gh/bobrenjc93/625/head -> origin/gh/bobrenjc93/625/head 2025-10-10T01:23:14.3271810Z * [new branch] gh/bobrenjc93/625/orig -> origin/gh/bobrenjc93/625/orig 2025-10-10T01:23:14.3274283Z * [new branch] gh/bobrenjc93/626/base -> origin/gh/bobrenjc93/626/base 2025-10-10T01:23:14.3276036Z * [new branch] gh/bobrenjc93/626/head -> origin/gh/bobrenjc93/626/head 2025-10-10T01:23:14.3277710Z * [new branch] gh/bobrenjc93/626/orig -> origin/gh/bobrenjc93/626/orig 2025-10-10T01:23:14.3280240Z * [new branch] gh/bobrenjc93/627/base -> origin/gh/bobrenjc93/627/base 2025-10-10T01:23:14.3281993Z * [new branch] gh/bobrenjc93/627/head -> origin/gh/bobrenjc93/627/head 2025-10-10T01:23:14.3283583Z * [new branch] gh/bobrenjc93/627/orig -> origin/gh/bobrenjc93/627/orig 2025-10-10T01:23:14.3286011Z * [new branch] gh/bobrenjc93/628/base -> origin/gh/bobrenjc93/628/base 2025-10-10T01:23:14.3287706Z * [new branch] gh/bobrenjc93/628/head -> origin/gh/bobrenjc93/628/head 2025-10-10T01:23:14.3289339Z * [new branch] gh/bobrenjc93/628/orig -> origin/gh/bobrenjc93/628/orig 2025-10-10T01:23:14.3291665Z * [new branch] gh/bobrenjc93/629/base -> origin/gh/bobrenjc93/629/base 2025-10-10T01:23:14.3293345Z * [new branch] gh/bobrenjc93/629/head -> origin/gh/bobrenjc93/629/head 2025-10-10T01:23:14.3295041Z * [new branch] gh/bobrenjc93/629/orig -> origin/gh/bobrenjc93/629/orig 2025-10-10T01:23:14.3299345Z * [new branch] gh/bobrenjc93/630/base -> origin/gh/bobrenjc93/630/base 2025-10-10T01:23:14.3301088Z * [new branch] gh/bobrenjc93/630/head -> origin/gh/bobrenjc93/630/head 2025-10-10T01:23:14.3302923Z * [new branch] gh/bobrenjc93/630/orig -> origin/gh/bobrenjc93/630/orig 2025-10-10T01:23:14.3305290Z * [new branch] gh/bobrenjc93/631/base -> origin/gh/bobrenjc93/631/base 2025-10-10T01:23:14.3306997Z * [new branch] gh/bobrenjc93/631/head -> origin/gh/bobrenjc93/631/head 2025-10-10T01:23:14.3308696Z * [new branch] gh/bobrenjc93/631/orig -> origin/gh/bobrenjc93/631/orig 2025-10-10T01:23:14.3311050Z * [new branch] gh/bobrenjc93/632/base -> origin/gh/bobrenjc93/632/base 2025-10-10T01:23:14.3312754Z * [new branch] gh/bobrenjc93/632/head -> origin/gh/bobrenjc93/632/head 2025-10-10T01:23:14.3314429Z * [new branch] gh/bobrenjc93/632/orig -> origin/gh/bobrenjc93/632/orig 2025-10-10T01:23:14.3316789Z * [new branch] gh/bobrenjc93/633/base -> origin/gh/bobrenjc93/633/base 2025-10-10T01:23:14.3318529Z * [new branch] gh/bobrenjc93/633/head -> origin/gh/bobrenjc93/633/head 2025-10-10T01:23:14.3320374Z * [new branch] gh/bobrenjc93/633/orig -> origin/gh/bobrenjc93/633/orig 2025-10-10T01:23:14.3322616Z * [new branch] gh/bobrenjc93/634/base -> origin/gh/bobrenjc93/634/base 2025-10-10T01:23:14.3324484Z * [new branch] gh/bobrenjc93/634/head -> origin/gh/bobrenjc93/634/head 2025-10-10T01:23:14.3326352Z * [new branch] gh/bobrenjc93/634/orig -> origin/gh/bobrenjc93/634/orig 2025-10-10T01:23:14.3328653Z * [new branch] gh/bobrenjc93/635/base -> origin/gh/bobrenjc93/635/base 2025-10-10T01:23:14.3330321Z * [new branch] gh/bobrenjc93/635/head -> origin/gh/bobrenjc93/635/head 2025-10-10T01:23:14.3332048Z * [new branch] gh/bobrenjc93/635/orig -> origin/gh/bobrenjc93/635/orig 2025-10-10T01:23:14.3334429Z * [new branch] gh/bobrenjc93/636/base -> origin/gh/bobrenjc93/636/base 2025-10-10T01:23:14.3336131Z * [new branch] gh/bobrenjc93/636/head -> origin/gh/bobrenjc93/636/head 2025-10-10T01:23:14.3337848Z * [new branch] gh/bobrenjc93/636/orig -> origin/gh/bobrenjc93/636/orig 2025-10-10T01:23:14.3340288Z * [new branch] gh/bobrenjc93/637/base -> origin/gh/bobrenjc93/637/base 2025-10-10T01:23:14.3341941Z * [new branch] gh/bobrenjc93/637/head -> origin/gh/bobrenjc93/637/head 2025-10-10T01:23:14.3343651Z * [new branch] gh/bobrenjc93/637/orig -> origin/gh/bobrenjc93/637/orig 2025-10-10T01:23:14.3346078Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-10-10T01:23:14.3348300Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-10-10T01:23:14.3350043Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-10-10T01:23:14.3352560Z * [new branch] gh/bobrenjc93/639/base -> origin/gh/bobrenjc93/639/base 2025-10-10T01:23:14.3354266Z * [new branch] gh/bobrenjc93/639/head -> origin/gh/bobrenjc93/639/head 2025-10-10T01:23:14.3356004Z * [new branch] gh/bobrenjc93/639/orig -> origin/gh/bobrenjc93/639/orig 2025-10-10T01:23:14.3358808Z * [new branch] gh/bobrenjc93/640/base -> origin/gh/bobrenjc93/640/base 2025-10-10T01:23:14.3360740Z * [new branch] gh/bobrenjc93/640/head -> origin/gh/bobrenjc93/640/head 2025-10-10T01:23:14.3362387Z * [new branch] gh/bobrenjc93/640/orig -> origin/gh/bobrenjc93/640/orig 2025-10-10T01:23:14.3364747Z * [new branch] gh/bobrenjc93/641/base -> origin/gh/bobrenjc93/641/base 2025-10-10T01:23:14.3366480Z * [new branch] gh/bobrenjc93/641/head -> origin/gh/bobrenjc93/641/head 2025-10-10T01:23:14.3368148Z * [new branch] gh/bobrenjc93/641/orig -> origin/gh/bobrenjc93/641/orig 2025-10-10T01:23:14.3370602Z * [new branch] gh/bobrenjc93/642/base -> origin/gh/bobrenjc93/642/base 2025-10-10T01:23:14.3372341Z * [new branch] gh/bobrenjc93/642/head -> origin/gh/bobrenjc93/642/head 2025-10-10T01:23:14.3374067Z * [new branch] gh/bobrenjc93/642/orig -> origin/gh/bobrenjc93/642/orig 2025-10-10T01:23:14.3376420Z * [new branch] gh/bobrenjc93/643/base -> origin/gh/bobrenjc93/643/base 2025-10-10T01:23:14.3378162Z * [new branch] gh/bobrenjc93/643/head -> origin/gh/bobrenjc93/643/head 2025-10-10T01:23:14.3379834Z * [new branch] gh/bobrenjc93/643/orig -> origin/gh/bobrenjc93/643/orig 2025-10-10T01:23:14.3382184Z * [new branch] gh/bobrenjc93/644/base -> origin/gh/bobrenjc93/644/base 2025-10-10T01:23:14.3383921Z * [new branch] gh/bobrenjc93/644/head -> origin/gh/bobrenjc93/644/head 2025-10-10T01:23:14.3385607Z * [new branch] gh/bobrenjc93/644/orig -> origin/gh/bobrenjc93/644/orig 2025-10-10T01:23:14.3387990Z * [new branch] gh/bobrenjc93/645/base -> origin/gh/bobrenjc93/645/base 2025-10-10T01:23:14.3389734Z * [new branch] gh/bobrenjc93/645/head -> origin/gh/bobrenjc93/645/head 2025-10-10T01:23:14.3391488Z * [new branch] gh/bobrenjc93/645/orig -> origin/gh/bobrenjc93/645/orig 2025-10-10T01:23:14.3393945Z * [new branch] gh/bobrenjc93/646/base -> origin/gh/bobrenjc93/646/base 2025-10-10T01:23:14.3395779Z * [new branch] gh/bobrenjc93/646/head -> origin/gh/bobrenjc93/646/head 2025-10-10T01:23:14.3397606Z * [new branch] gh/bobrenjc93/646/orig -> origin/gh/bobrenjc93/646/orig 2025-10-10T01:23:14.3400304Z * [new branch] gh/bobrenjc93/647/base -> origin/gh/bobrenjc93/647/base 2025-10-10T01:23:14.3402019Z * [new branch] gh/bobrenjc93/647/head -> origin/gh/bobrenjc93/647/head 2025-10-10T01:23:14.3404159Z * [new branch] gh/bobrenjc93/647/orig -> origin/gh/bobrenjc93/647/orig 2025-10-10T01:23:14.3406338Z * [new branch] gh/bobrenjc93/648/base -> origin/gh/bobrenjc93/648/base 2025-10-10T01:23:14.3408025Z * [new branch] gh/bobrenjc93/648/head -> origin/gh/bobrenjc93/648/head 2025-10-10T01:23:14.3409445Z * [new branch] gh/bobrenjc93/648/orig -> origin/gh/bobrenjc93/648/orig 2025-10-10T01:23:14.3412164Z * [new branch] gh/bobrenjc93/649/base -> origin/gh/bobrenjc93/649/base 2025-10-10T01:23:14.3413339Z * [new branch] gh/bobrenjc93/649/head -> origin/gh/bobrenjc93/649/head 2025-10-10T01:23:14.3415147Z * [new branch] gh/bobrenjc93/649/orig -> origin/gh/bobrenjc93/649/orig 2025-10-10T01:23:14.3417667Z * [new branch] gh/bobrenjc93/650/base -> origin/gh/bobrenjc93/650/base 2025-10-10T01:23:14.3419294Z * [new branch] gh/bobrenjc93/650/head -> origin/gh/bobrenjc93/650/head 2025-10-10T01:23:14.3420989Z * [new branch] gh/bobrenjc93/650/orig -> origin/gh/bobrenjc93/650/orig 2025-10-10T01:23:14.3423898Z * [new branch] gh/briancoutinho/2/base -> origin/gh/briancoutinho/2/base 2025-10-10T01:23:14.3425758Z * [new branch] gh/briancoutinho/2/head -> origin/gh/briancoutinho/2/head 2025-10-10T01:23:14.3428655Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-10-10T01:23:14.3430328Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-10-10T01:23:14.3432841Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-10-10T01:23:14.3434435Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-10-10T01:23:14.3436102Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-10-10T01:23:14.3438352Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-10-10T01:23:14.3440207Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-10-10T01:23:14.3441955Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-10-10T01:23:14.3444585Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-10-10T01:23:14.3446389Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-10-10T01:23:14.3448224Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-10-10T01:23:14.3451053Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-10-10T01:23:14.3452884Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-10-10T01:23:14.3454583Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-10-10T01:23:14.3457681Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-10-10T01:23:14.3459057Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-10-10T01:23:14.3461570Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-10-10T01:23:14.3463451Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-10-10T01:23:14.3465184Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-10-10T01:23:14.3467749Z * [new branch] gh/coconutruben/22/base -> origin/gh/coconutruben/22/base 2025-10-10T01:23:14.3469260Z * [new branch] gh/coconutruben/22/head -> origin/gh/coconutruben/22/head 2025-10-10T01:23:14.3470959Z * [new branch] gh/coconutruben/22/orig -> origin/gh/coconutruben/22/orig 2025-10-10T01:23:14.3473491Z * [new branch] gh/coconutruben/24/base -> origin/gh/coconutruben/24/base 2025-10-10T01:23:14.3475298Z * [new branch] gh/coconutruben/24/head -> origin/gh/coconutruben/24/head 2025-10-10T01:23:14.3477041Z * [new branch] gh/coconutruben/24/orig -> origin/gh/coconutruben/24/orig 2025-10-10T01:23:14.3479857Z * [new branch] gh/coconutruben/25/base -> origin/gh/coconutruben/25/base 2025-10-10T01:23:14.3481882Z * [new branch] gh/coconutruben/25/head -> origin/gh/coconutruben/25/head 2025-10-10T01:23:14.3483739Z * [new branch] gh/coconutruben/25/orig -> origin/gh/coconutruben/25/orig 2025-10-10T01:23:14.3487280Z * [new branch] gh/coconutruben/36/base -> origin/gh/coconutruben/36/base 2025-10-10T01:23:14.3489398Z * [new branch] gh/coconutruben/36/head -> origin/gh/coconutruben/36/head 2025-10-10T01:23:14.3491849Z * [new branch] gh/coconutruben/36/orig -> origin/gh/coconutruben/36/orig 2025-10-10T01:23:14.3494261Z * [new branch] gh/coconutruben/48/base -> origin/gh/coconutruben/48/base 2025-10-10T01:23:14.3496047Z * [new branch] gh/coconutruben/48/head -> origin/gh/coconutruben/48/head 2025-10-10T01:23:14.3498084Z * [new branch] gh/coconutruben/48/orig -> origin/gh/coconutruben/48/orig 2025-10-10T01:23:14.3500431Z * [new branch] gh/coconutruben/49/base -> origin/gh/coconutruben/49/base 2025-10-10T01:23:14.3502151Z * [new branch] gh/coconutruben/49/head -> origin/gh/coconutruben/49/head 2025-10-10T01:23:14.3503945Z * [new branch] gh/coconutruben/49/orig -> origin/gh/coconutruben/49/orig 2025-10-10T01:23:14.3506430Z * [new branch] gh/coconutruben/50/base -> origin/gh/coconutruben/50/base 2025-10-10T01:23:14.3508719Z * [new branch] gh/coconutruben/50/head -> origin/gh/coconutruben/50/head 2025-10-10T01:23:14.3510578Z * [new branch] gh/coconutruben/50/orig -> origin/gh/coconutruben/50/orig 2025-10-10T01:23:14.3513042Z * [new branch] gh/coconutruben/51/base -> origin/gh/coconutruben/51/base 2025-10-10T01:23:14.3514820Z * [new branch] gh/coconutruben/51/head -> origin/gh/coconutruben/51/head 2025-10-10T01:23:14.3516573Z * [new branch] gh/coconutruben/51/orig -> origin/gh/coconutruben/51/orig 2025-10-10T01:23:14.3519640Z * [new branch] gh/coconutruben/52/base -> origin/gh/coconutruben/52/base 2025-10-10T01:23:14.3521576Z * [new branch] gh/coconutruben/52/head -> origin/gh/coconutruben/52/head 2025-10-10T01:23:14.3523302Z * [new branch] gh/coconutruben/52/orig -> origin/gh/coconutruben/52/orig 2025-10-10T01:23:14.3525721Z * [new branch] gh/coconutruben/53/base -> origin/gh/coconutruben/53/base 2025-10-10T01:23:14.3527517Z * [new branch] gh/coconutruben/53/head -> origin/gh/coconutruben/53/head 2025-10-10T01:23:14.3529705Z * [new branch] gh/coconutruben/53/orig -> origin/gh/coconutruben/53/orig 2025-10-10T01:23:14.3532122Z * [new branch] gh/coconutruben/54/base -> origin/gh/coconutruben/54/base 2025-10-10T01:23:14.3534028Z * [new branch] gh/coconutruben/54/head -> origin/gh/coconutruben/54/head 2025-10-10T01:23:14.3535802Z * [new branch] gh/coconutruben/54/orig -> origin/gh/coconutruben/54/orig 2025-10-10T01:23:14.3538385Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-10-10T01:23:14.3540189Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-10-10T01:23:14.3541863Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-10-10T01:23:14.3544409Z * [new branch] gh/coconutruben/56/base -> origin/gh/coconutruben/56/base 2025-10-10T01:23:14.3546121Z * [new branch] gh/coconutruben/56/head -> origin/gh/coconutruben/56/head 2025-10-10T01:23:14.3547951Z * [new branch] gh/coconutruben/56/orig -> origin/gh/coconutruben/56/orig 2025-10-10T01:23:14.3550421Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-10-10T01:23:14.3552171Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-10-10T01:23:14.3553936Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-10-10T01:23:14.3556597Z * [new branch] gh/coconutruben/58/base -> origin/gh/coconutruben/58/base 2025-10-10T01:23:14.3558415Z * [new branch] gh/coconutruben/58/head -> origin/gh/coconutruben/58/head 2025-10-10T01:23:14.3560251Z * [new branch] gh/coconutruben/58/orig -> origin/gh/coconutruben/58/orig 2025-10-10T01:23:14.3562698Z * [new branch] gh/coconutruben/59/base -> origin/gh/coconutruben/59/base 2025-10-10T01:23:14.3564430Z * [new branch] gh/coconutruben/59/head -> origin/gh/coconutruben/59/head 2025-10-10T01:23:14.3566136Z * [new branch] gh/coconutruben/59/orig -> origin/gh/coconutruben/59/orig 2025-10-10T01:23:14.3569194Z * [new branch] gh/coconutruben/62/base -> origin/gh/coconutruben/62/base 2025-10-10T01:23:14.3570989Z * [new branch] gh/coconutruben/62/head -> origin/gh/coconutruben/62/head 2025-10-10T01:23:14.3572861Z * [new branch] gh/coconutruben/62/orig -> origin/gh/coconutruben/62/orig 2025-10-10T01:23:14.3575341Z * [new branch] gh/coconutruben/64/base -> origin/gh/coconutruben/64/base 2025-10-10T01:23:14.3577140Z * [new branch] gh/coconutruben/64/head -> origin/gh/coconutruben/64/head 2025-10-10T01:23:14.3578806Z * [new branch] gh/coconutruben/64/orig -> origin/gh/coconutruben/64/orig 2025-10-10T01:23:14.3581205Z * [new branch] gh/coconutruben/65/base -> origin/gh/coconutruben/65/base 2025-10-10T01:23:14.3583169Z * [new branch] gh/coconutruben/65/head -> origin/gh/coconutruben/65/head 2025-10-10T01:23:14.3584840Z * [new branch] gh/coconutruben/65/orig -> origin/gh/coconutruben/65/orig 2025-10-10T01:23:14.3587375Z * [new branch] gh/coconutruben/66/base -> origin/gh/coconutruben/66/base 2025-10-10T01:23:14.3589122Z * [new branch] gh/coconutruben/66/head -> origin/gh/coconutruben/66/head 2025-10-10T01:23:14.3590952Z * [new branch] gh/coconutruben/66/orig -> origin/gh/coconutruben/66/orig 2025-10-10T01:23:14.3593425Z * [new branch] gh/coconutruben/67/base -> origin/gh/coconutruben/67/base 2025-10-10T01:23:14.3595247Z * [new branch] gh/coconutruben/67/head -> origin/gh/coconutruben/67/head 2025-10-10T01:23:14.3597202Z * [new branch] gh/coconutruben/67/orig -> origin/gh/coconutruben/67/orig 2025-10-10T01:23:14.3599806Z * [new branch] gh/coconutruben/68/base -> origin/gh/coconutruben/68/base 2025-10-10T01:23:14.3601470Z * [new branch] gh/coconutruben/68/head -> origin/gh/coconutruben/68/head 2025-10-10T01:23:14.3603223Z * [new branch] gh/coconutruben/68/orig -> origin/gh/coconutruben/68/orig 2025-10-10T01:23:14.3605646Z * [new branch] gh/coconutruben/69/base -> origin/gh/coconutruben/69/base 2025-10-10T01:23:14.3607343Z * [new branch] gh/coconutruben/69/head -> origin/gh/coconutruben/69/head 2025-10-10T01:23:14.3609250Z * [new branch] gh/coconutruben/69/orig -> origin/gh/coconutruben/69/orig 2025-10-10T01:23:14.3611536Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-10-10T01:23:14.3613333Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-10-10T01:23:14.3615086Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-10-10T01:23:14.3617360Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-10-10T01:23:14.3619076Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-10-10T01:23:14.3620779Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-10-10T01:23:14.3623149Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-10-10T01:23:14.3624914Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-10-10T01:23:14.3626659Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-10-10T01:23:14.3628930Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-10-10T01:23:14.3630678Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-10-10T01:23:14.3632329Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-10-10T01:23:14.3634885Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-10-10T01:23:14.3636692Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-10-10T01:23:14.3638410Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-10-10T01:23:14.3641008Z * [new branch] gh/coconutruben/75/base -> origin/gh/coconutruben/75/base 2025-10-10T01:23:14.3642733Z * [new branch] gh/coconutruben/75/head -> origin/gh/coconutruben/75/head 2025-10-10T01:23:14.3644551Z * [new branch] gh/coconutruben/75/orig -> origin/gh/coconutruben/75/orig 2025-10-10T01:23:14.3647124Z * [new branch] gh/coconutruben/76/base -> origin/gh/coconutruben/76/base 2025-10-10T01:23:14.3649010Z * [new branch] gh/coconutruben/76/head -> origin/gh/coconutruben/76/head 2025-10-10T01:23:14.3650817Z * [new branch] gh/coconutruben/76/orig -> origin/gh/coconutruben/76/orig 2025-10-10T01:23:14.3653361Z * [new branch] gh/coconutruben/77/base -> origin/gh/coconutruben/77/base 2025-10-10T01:23:14.3655123Z * [new branch] gh/coconutruben/77/head -> origin/gh/coconutruben/77/head 2025-10-10T01:23:14.3656836Z * [new branch] gh/coconutruben/77/orig -> origin/gh/coconutruben/77/orig 2025-10-10T01:23:14.3659414Z * [new branch] gh/coconutruben/78/base -> origin/gh/coconutruben/78/base 2025-10-10T01:23:14.3661151Z * [new branch] gh/coconutruben/78/head -> origin/gh/coconutruben/78/head 2025-10-10T01:23:14.3662849Z * [new branch] gh/coconutruben/78/orig -> origin/gh/coconutruben/78/orig 2025-10-10T01:23:14.3665282Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-10-10T01:23:14.3667105Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-10-10T01:23:14.3668825Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-10-10T01:23:14.3671202Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-10-10T01:23:14.3672975Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-10-10T01:23:14.3674739Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-10-10T01:23:14.3677269Z * [new branch] gh/coconutruben/81/base -> origin/gh/coconutruben/81/base 2025-10-10T01:23:14.3678943Z * [new branch] gh/coconutruben/81/head -> origin/gh/coconutruben/81/head 2025-10-10T01:23:14.3680746Z * [new branch] gh/coconutruben/81/orig -> origin/gh/coconutruben/81/orig 2025-10-10T01:23:14.3683068Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-10-10T01:23:14.3684790Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-10-10T01:23:14.3686529Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-10-10T01:23:14.3688829Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-10-10T01:23:14.3690536Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-10-10T01:23:14.3692206Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-10-10T01:23:14.3695111Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-10-10T01:23:14.3697390Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-10-10T01:23:14.3700489Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-10-10T01:23:14.3702122Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-10-10T01:23:14.3704448Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-10-10T01:23:14.3706060Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-10-10T01:23:14.3708364Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-10-10T01:23:14.3710024Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-10-10T01:23:14.3712898Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-10-10T01:23:14.3714695Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-10-10T01:23:14.3716403Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-10-10T01:23:14.3718745Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-10-10T01:23:14.3720716Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-10-10T01:23:14.3722420Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-10-10T01:23:14.3725585Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-10-10T01:23:14.3728128Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-10-10T01:23:14.3729902Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-10-10T01:23:14.3732205Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-10-10T01:23:14.3733893Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-10-10T01:23:14.3735756Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-10-10T01:23:14.3738983Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-10-10T01:23:14.3740809Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-10-10T01:23:14.3742440Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-10-10T01:23:14.3744878Z * [new branch] gh/davidberard98/401/base -> origin/gh/davidberard98/401/base 2025-10-10T01:23:14.3746651Z * [new branch] gh/davidberard98/401/head -> origin/gh/davidberard98/401/head 2025-10-10T01:23:14.3749364Z * [new branch] gh/davidberard98/401/orig -> origin/gh/davidberard98/401/orig 2025-10-10T01:23:14.3750692Z * [new branch] gh/davidberard98/405/base -> origin/gh/davidberard98/405/base 2025-10-10T01:23:14.3752481Z * [new branch] gh/davidberard98/405/head -> origin/gh/davidberard98/405/head 2025-10-10T01:23:14.3754181Z * [new branch] gh/davidberard98/405/orig -> origin/gh/davidberard98/405/orig 2025-10-10T01:23:14.3756490Z * [new branch] gh/davidberard98/410/base -> origin/gh/davidberard98/410/base 2025-10-10T01:23:14.3758359Z * [new branch] gh/davidberard98/410/head -> origin/gh/davidberard98/410/head 2025-10-10T01:23:14.3760050Z * [new branch] gh/davidberard98/410/orig -> origin/gh/davidberard98/410/orig 2025-10-10T01:23:14.3762322Z * [new branch] gh/davidberard98/411/base -> origin/gh/davidberard98/411/base 2025-10-10T01:23:14.3764056Z * [new branch] gh/davidberard98/411/head -> origin/gh/davidberard98/411/head 2025-10-10T01:23:14.3765727Z * [new branch] gh/davidberard98/411/orig -> origin/gh/davidberard98/411/orig 2025-10-10T01:23:14.3768021Z * [new branch] gh/davidberard98/412/base -> origin/gh/davidberard98/412/base 2025-10-10T01:23:14.3770767Z * [new branch] gh/davidberard98/412/head -> origin/gh/davidberard98/412/head 2025-10-10T01:23:14.3772677Z * [new branch] gh/davidberard98/412/orig -> origin/gh/davidberard98/412/orig 2025-10-10T01:23:14.3775544Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-10-10T01:23:14.3777275Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-10-10T01:23:14.3779014Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-10-10T01:23:14.3781239Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-10-10T01:23:14.3782989Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-10-10T01:23:14.3784707Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-10-10T01:23:14.3787433Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-10-10T01:23:14.3789473Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-10-10T01:23:14.3790792Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-10-10T01:23:14.3793173Z * [new branch] gh/desertfire/598/base -> origin/gh/desertfire/598/base 2025-10-10T01:23:14.3794892Z * [new branch] gh/desertfire/598/head -> origin/gh/desertfire/598/head 2025-10-10T01:23:14.3796820Z * [new branch] gh/desertfire/598/orig -> origin/gh/desertfire/598/orig 2025-10-10T01:23:14.3799856Z * [new branch] gh/desertfire/599/base -> origin/gh/desertfire/599/base 2025-10-10T01:23:14.3801847Z * [new branch] gh/desertfire/599/head -> origin/gh/desertfire/599/head 2025-10-10T01:23:14.3804202Z * [new branch] gh/desertfire/599/orig -> origin/gh/desertfire/599/orig 2025-10-10T01:23:14.3806924Z * [new branch] gh/desertfire/600/base -> origin/gh/desertfire/600/base 2025-10-10T01:23:14.3808680Z * [new branch] gh/desertfire/600/head -> origin/gh/desertfire/600/head 2025-10-10T01:23:14.3810481Z * [new branch] gh/desertfire/600/orig -> origin/gh/desertfire/600/orig 2025-10-10T01:23:14.3812925Z * [new branch] gh/desertfire/601/base -> origin/gh/desertfire/601/base 2025-10-10T01:23:14.3814679Z * [new branch] gh/desertfire/601/head -> origin/gh/desertfire/601/head 2025-10-10T01:23:14.3816386Z * [new branch] gh/desertfire/601/orig -> origin/gh/desertfire/601/orig 2025-10-10T01:23:14.3819206Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-10-10T01:23:14.3821281Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-10-10T01:23:14.3823986Z * [new branch] gh/drisspg/159/base -> origin/gh/drisspg/159/base 2025-10-10T01:23:14.3825636Z * [new branch] gh/drisspg/159/head -> origin/gh/drisspg/159/head 2025-10-10T01:23:14.3827321Z * [new branch] gh/drisspg/159/orig -> origin/gh/drisspg/159/orig 2025-10-10T01:23:14.3829663Z * [new branch] gh/drisspg/166/base -> origin/gh/drisspg/166/base 2025-10-10T01:23:14.3831383Z * [new branch] gh/drisspg/166/head -> origin/gh/drisspg/166/head 2025-10-10T01:23:14.3833545Z * [new branch] gh/drisspg/166/orig -> origin/gh/drisspg/166/orig 2025-10-10T01:23:14.3835874Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-10-10T01:23:14.3838029Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-10-10T01:23:14.3839737Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-10-10T01:23:14.3842400Z * [new branch] gh/drisspg/177/base -> origin/gh/drisspg/177/base 2025-10-10T01:23:14.3843722Z * [new branch] gh/drisspg/177/head -> origin/gh/drisspg/177/head 2025-10-10T01:23:14.3845574Z * [new branch] gh/drisspg/177/orig -> origin/gh/drisspg/177/orig 2025-10-10T01:23:14.3848011Z * [new branch] gh/drisspg/178/base -> origin/gh/drisspg/178/base 2025-10-10T01:23:14.3849621Z * [new branch] gh/drisspg/178/head -> origin/gh/drisspg/178/head 2025-10-10T01:23:14.3851186Z * [new branch] gh/drisspg/178/orig -> origin/gh/drisspg/178/orig 2025-10-10T01:23:14.3853555Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-10-10T01:23:14.3855262Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-10-10T01:23:14.3857412Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-10-10T01:23:14.3859041Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-10-10T01:23:14.3861146Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-10-10T01:23:14.3862716Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-10-10T01:23:14.3865080Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-10-10T01:23:14.3866753Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-10-10T01:23:14.3869167Z * [new branch] gh/drisspg/187/base -> origin/gh/drisspg/187/base 2025-10-10T01:23:14.3871137Z * [new branch] gh/drisspg/187/head -> origin/gh/drisspg/187/head 2025-10-10T01:23:14.3872975Z * [new branch] gh/drisspg/187/orig -> origin/gh/drisspg/187/orig 2025-10-10T01:23:14.3875248Z * [new branch] gh/drisspg/188/base -> origin/gh/drisspg/188/base 2025-10-10T01:23:14.3876916Z * [new branch] gh/drisspg/188/head -> origin/gh/drisspg/188/head 2025-10-10T01:23:14.3878692Z * [new branch] gh/drisspg/188/orig -> origin/gh/drisspg/188/orig 2025-10-10T01:23:14.3881259Z * [new branch] gh/drisspg/189/base -> origin/gh/drisspg/189/base 2025-10-10T01:23:14.3882883Z * [new branch] gh/drisspg/189/head -> origin/gh/drisspg/189/head 2025-10-10T01:23:14.3884523Z * [new branch] gh/drisspg/189/orig -> origin/gh/drisspg/189/orig 2025-10-10T01:23:14.3886890Z * [new branch] gh/drisspg/193/base -> origin/gh/drisspg/193/base 2025-10-10T01:23:14.3888525Z * [new branch] gh/drisspg/193/head -> origin/gh/drisspg/193/head 2025-10-10T01:23:14.3890456Z * [new branch] gh/drisspg/193/orig -> origin/gh/drisspg/193/orig 2025-10-10T01:23:14.3893197Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-10-10T01:23:14.3894385Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-10-10T01:23:14.3896018Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-10-10T01:23:14.3898723Z * [new branch] gh/drisspg/196/base -> origin/gh/drisspg/196/base 2025-10-10T01:23:14.3900399Z * [new branch] gh/drisspg/196/head -> origin/gh/drisspg/196/head 2025-10-10T01:23:14.3902044Z * [new branch] gh/drisspg/196/orig -> origin/gh/drisspg/196/orig 2025-10-10T01:23:14.3904321Z * [new branch] gh/drisspg/197/base -> origin/gh/drisspg/197/base 2025-10-10T01:23:14.3906013Z * [new branch] gh/drisspg/197/head -> origin/gh/drisspg/197/head 2025-10-10T01:23:14.3907676Z * [new branch] gh/drisspg/197/orig -> origin/gh/drisspg/197/orig 2025-10-10T01:23:14.3910465Z * [new branch] gh/drisspg/198/base -> origin/gh/drisspg/198/base 2025-10-10T01:23:14.3912345Z * [new branch] gh/drisspg/198/head -> origin/gh/drisspg/198/head 2025-10-10T01:23:14.3913941Z * [new branch] gh/drisspg/198/orig -> origin/gh/drisspg/198/orig 2025-10-10T01:23:14.3916385Z * [new branch] gh/drisspg/199/base -> origin/gh/drisspg/199/base 2025-10-10T01:23:14.3918053Z * [new branch] gh/drisspg/199/head -> origin/gh/drisspg/199/head 2025-10-10T01:23:14.3919914Z * [new branch] gh/drisspg/199/orig -> origin/gh/drisspg/199/orig 2025-10-10T01:23:14.3922342Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-10-10T01:23:14.3924020Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-10-10T01:23:14.3925704Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-10-10T01:23:14.3928086Z * [new branch] gh/drisspg/201/base -> origin/gh/drisspg/201/base 2025-10-10T01:23:14.3929779Z * [new branch] gh/drisspg/201/head -> origin/gh/drisspg/201/head 2025-10-10T01:23:14.3931452Z * [new branch] gh/drisspg/201/orig -> origin/gh/drisspg/201/orig 2025-10-10T01:23:14.3933795Z * [new branch] gh/drisspg/202/base -> origin/gh/drisspg/202/base 2025-10-10T01:23:14.3935463Z * [new branch] gh/drisspg/202/head -> origin/gh/drisspg/202/head 2025-10-10T01:23:14.3937146Z * [new branch] gh/drisspg/202/orig -> origin/gh/drisspg/202/orig 2025-10-10T01:23:14.3939575Z * [new branch] gh/drisspg/203/base -> origin/gh/drisspg/203/base 2025-10-10T01:23:14.3941385Z * [new branch] gh/drisspg/203/head -> origin/gh/drisspg/203/head 2025-10-10T01:23:14.3943004Z * [new branch] gh/drisspg/203/orig -> origin/gh/drisspg/203/orig 2025-10-10T01:23:14.3945424Z * [new branch] gh/drisspg/204/base -> origin/gh/drisspg/204/base 2025-10-10T01:23:14.3947147Z * [new branch] gh/drisspg/204/head -> origin/gh/drisspg/204/head 2025-10-10T01:23:14.3948883Z * [new branch] gh/drisspg/204/orig -> origin/gh/drisspg/204/orig 2025-10-10T01:23:14.3951295Z * [new branch] gh/drisspg/205/base -> origin/gh/drisspg/205/base 2025-10-10T01:23:14.3953001Z * [new branch] gh/drisspg/205/head -> origin/gh/drisspg/205/head 2025-10-10T01:23:14.3954716Z * [new branch] gh/drisspg/205/orig -> origin/gh/drisspg/205/orig 2025-10-10T01:23:14.3957066Z * [new branch] gh/drisspg/206/base -> origin/gh/drisspg/206/base 2025-10-10T01:23:14.3958925Z * [new branch] gh/drisspg/206/head -> origin/gh/drisspg/206/head 2025-10-10T01:23:14.3960662Z * [new branch] gh/drisspg/206/orig -> origin/gh/drisspg/206/orig 2025-10-10T01:23:14.3963173Z * [new branch] gh/drisspg/207/base -> origin/gh/drisspg/207/base 2025-10-10T01:23:14.3964828Z * [new branch] gh/drisspg/207/head -> origin/gh/drisspg/207/head 2025-10-10T01:23:14.3966501Z * [new branch] gh/drisspg/207/orig -> origin/gh/drisspg/207/orig 2025-10-10T01:23:14.3968863Z * [new branch] gh/drisspg/208/base -> origin/gh/drisspg/208/base 2025-10-10T01:23:14.3970585Z * [new branch] gh/drisspg/208/head -> origin/gh/drisspg/208/head 2025-10-10T01:23:14.3972217Z * [new branch] gh/drisspg/208/orig -> origin/gh/drisspg/208/orig 2025-10-10T01:23:14.3974514Z * [new branch] gh/drisspg/209/base -> origin/gh/drisspg/209/base 2025-10-10T01:23:14.3976206Z * [new branch] gh/drisspg/209/head -> origin/gh/drisspg/209/head 2025-10-10T01:23:14.3978019Z * [new branch] gh/drisspg/209/orig -> origin/gh/drisspg/209/orig 2025-10-10T01:23:14.3980871Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-10-10T01:23:14.3982650Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-10-10T01:23:14.3985605Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-10-10T01:23:14.3987367Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-10-10T01:23:14.3989050Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-10-10T01:23:14.3991998Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-10-10T01:23:14.3993787Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-10-10T01:23:14.3995504Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-10-10T01:23:14.3998078Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-10-10T01:23:14.3999894Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-10-10T01:23:14.4001561Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-10-10T01:23:14.4004297Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-10-10T01:23:14.4005981Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-10-10T01:23:14.4007714Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-10-10T01:23:14.4010088Z * [new branch] gh/eellison/824/base -> origin/gh/eellison/824/base 2025-10-10T01:23:14.4011783Z * [new branch] gh/eellison/824/head -> origin/gh/eellison/824/head 2025-10-10T01:23:14.4013451Z * [new branch] gh/eellison/824/orig -> origin/gh/eellison/824/orig 2025-10-10T01:23:14.4015932Z * [new branch] gh/eellison/825/base -> origin/gh/eellison/825/base 2025-10-10T01:23:14.4017624Z * [new branch] gh/eellison/825/head -> origin/gh/eellison/825/head 2025-10-10T01:23:14.4019769Z * [new branch] gh/eellison/825/orig -> origin/gh/eellison/825/orig 2025-10-10T01:23:14.4022064Z * [new branch] gh/eellison/826/base -> origin/gh/eellison/826/base 2025-10-10T01:23:14.4023978Z * [new branch] gh/eellison/826/head -> origin/gh/eellison/826/head 2025-10-10T01:23:14.4025579Z * [new branch] gh/eellison/826/orig -> origin/gh/eellison/826/orig 2025-10-10T01:23:14.4027819Z * [new branch] gh/eellison/827/base -> origin/gh/eellison/827/base 2025-10-10T01:23:14.4029690Z * [new branch] gh/eellison/827/head -> origin/gh/eellison/827/head 2025-10-10T01:23:14.4031306Z * [new branch] gh/eellison/827/orig -> origin/gh/eellison/827/orig 2025-10-10T01:23:14.4033552Z * [new branch] gh/eellison/828/base -> origin/gh/eellison/828/base 2025-10-10T01:23:14.4035223Z * [new branch] gh/eellison/828/head -> origin/gh/eellison/828/head 2025-10-10T01:23:14.4036844Z * [new branch] gh/eellison/828/orig -> origin/gh/eellison/828/orig 2025-10-10T01:23:14.4039426Z * [new branch] gh/eellison/829/base -> origin/gh/eellison/829/base 2025-10-10T01:23:14.4041243Z * [new branch] gh/eellison/829/head -> origin/gh/eellison/829/head 2025-10-10T01:23:14.4042933Z * [new branch] gh/eellison/829/orig -> origin/gh/eellison/829/orig 2025-10-10T01:23:14.4045860Z * [new branch] gh/eellison/830/base -> origin/gh/eellison/830/base 2025-10-10T01:23:14.4047658Z * [new branch] gh/eellison/830/head -> origin/gh/eellison/830/head 2025-10-10T01:23:14.4049366Z * [new branch] gh/eellison/830/orig -> origin/gh/eellison/830/orig 2025-10-10T01:23:14.4051646Z * [new branch] gh/eellison/831/base -> origin/gh/eellison/831/base 2025-10-10T01:23:14.4053443Z * [new branch] gh/eellison/831/head -> origin/gh/eellison/831/head 2025-10-10T01:23:14.4055134Z * [new branch] gh/eellison/831/orig -> origin/gh/eellison/831/orig 2025-10-10T01:23:14.4057408Z * [new branch] gh/eellison/832/base -> origin/gh/eellison/832/base 2025-10-10T01:23:14.4059089Z * [new branch] gh/eellison/832/head -> origin/gh/eellison/832/head 2025-10-10T01:23:14.4060774Z * [new branch] gh/eellison/832/orig -> origin/gh/eellison/832/orig 2025-10-10T01:23:14.4063159Z * [new branch] gh/eellison/833/base -> origin/gh/eellison/833/base 2025-10-10T01:23:14.4064885Z * [new branch] gh/eellison/833/head -> origin/gh/eellison/833/head 2025-10-10T01:23:14.4066591Z * [new branch] gh/eellison/833/orig -> origin/gh/eellison/833/orig 2025-10-10T01:23:14.4068869Z * [new branch] gh/eellison/834/base -> origin/gh/eellison/834/base 2025-10-10T01:23:14.4070594Z * [new branch] gh/eellison/834/head -> origin/gh/eellison/834/head 2025-10-10T01:23:14.4072178Z * [new branch] gh/eellison/834/orig -> origin/gh/eellison/834/orig 2025-10-10T01:23:14.4074482Z * [new branch] gh/eellison/835/base -> origin/gh/eellison/835/base 2025-10-10T01:23:14.4076215Z * [new branch] gh/eellison/835/head -> origin/gh/eellison/835/head 2025-10-10T01:23:14.4077959Z * [new branch] gh/eellison/835/orig -> origin/gh/eellison/835/orig 2025-10-10T01:23:14.4081058Z * [new branch] gh/eellison/836/base -> origin/gh/eellison/836/base 2025-10-10T01:23:14.4082770Z * [new branch] gh/eellison/836/head -> origin/gh/eellison/836/head 2025-10-10T01:23:14.4084543Z * [new branch] gh/eellison/836/orig -> origin/gh/eellison/836/orig 2025-10-10T01:23:14.4087066Z * [new branch] gh/eellison/837/base -> origin/gh/eellison/837/base 2025-10-10T01:23:14.4088749Z * [new branch] gh/eellison/837/head -> origin/gh/eellison/837/head 2025-10-10T01:23:14.4090471Z * [new branch] gh/eellison/837/orig -> origin/gh/eellison/837/orig 2025-10-10T01:23:14.4092860Z * [new branch] gh/eellison/838/base -> origin/gh/eellison/838/base 2025-10-10T01:23:14.4094565Z * [new branch] gh/eellison/838/head -> origin/gh/eellison/838/head 2025-10-10T01:23:14.4096569Z * [new branch] gh/eellison/838/orig -> origin/gh/eellison/838/orig 2025-10-10T01:23:14.4100853Z * [new branch] gh/eellison/839/base -> origin/gh/eellison/839/base 2025-10-10T01:23:14.4102491Z * [new branch] gh/eellison/839/head -> origin/gh/eellison/839/head 2025-10-10T01:23:14.4104340Z * [new branch] gh/eellison/839/orig -> origin/gh/eellison/839/orig 2025-10-10T01:23:14.4106723Z * [new branch] gh/eellison/840/base -> origin/gh/eellison/840/base 2025-10-10T01:23:14.4108457Z * [new branch] gh/eellison/840/head -> origin/gh/eellison/840/head 2025-10-10T01:23:14.4110131Z * [new branch] gh/eellison/840/orig -> origin/gh/eellison/840/orig 2025-10-10T01:23:14.4112486Z * [new branch] gh/eellison/841/base -> origin/gh/eellison/841/base 2025-10-10T01:23:14.4114634Z * [new branch] gh/eellison/841/head -> origin/gh/eellison/841/head 2025-10-10T01:23:14.4116306Z * [new branch] gh/eellison/841/orig -> origin/gh/eellison/841/orig 2025-10-10T01:23:14.4118565Z * [new branch] gh/eellison/842/base -> origin/gh/eellison/842/base 2025-10-10T01:23:14.4120510Z * [new branch] gh/eellison/842/head -> origin/gh/eellison/842/head 2025-10-10T01:23:14.4122262Z * [new branch] gh/eellison/842/orig -> origin/gh/eellison/842/orig 2025-10-10T01:23:14.4124685Z * [new branch] gh/eellison/843/base -> origin/gh/eellison/843/base 2025-10-10T01:23:14.4126319Z * [new branch] gh/eellison/843/head -> origin/gh/eellison/843/head 2025-10-10T01:23:14.4128081Z * [new branch] gh/eellison/843/orig -> origin/gh/eellison/843/orig 2025-10-10T01:23:14.4130339Z * [new branch] gh/eellison/844/base -> origin/gh/eellison/844/base 2025-10-10T01:23:14.4132074Z * [new branch] gh/eellison/844/head -> origin/gh/eellison/844/head 2025-10-10T01:23:14.4133794Z * [new branch] gh/eellison/844/orig -> origin/gh/eellison/844/orig 2025-10-10T01:23:14.4136282Z * [new branch] gh/eellison/845/base -> origin/gh/eellison/845/base 2025-10-10T01:23:14.4137948Z * [new branch] gh/eellison/845/head -> origin/gh/eellison/845/head 2025-10-10T01:23:14.4139659Z * [new branch] gh/eellison/845/orig -> origin/gh/eellison/845/orig 2025-10-10T01:23:14.4142022Z * [new branch] gh/eellison/846/base -> origin/gh/eellison/846/base 2025-10-10T01:23:14.4143738Z * [new branch] gh/eellison/846/head -> origin/gh/eellison/846/head 2025-10-10T01:23:14.4145404Z * [new branch] gh/eellison/846/orig -> origin/gh/eellison/846/orig 2025-10-10T01:23:14.4148287Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-10-10T01:23:14.4150052Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-10-10T01:23:14.4152600Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-10-10T01:23:14.4154366Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-10-10T01:23:14.4156042Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-10-10T01:23:14.4158293Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-10-10T01:23:14.4160381Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-10-10T01:23:14.4162496Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-10-10T01:23:14.4166522Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-10-10T01:23:14.4168946Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-10-10T01:23:14.4171304Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-10-10T01:23:14.4174462Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-10-10T01:23:14.4176248Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-10-10T01:23:14.4177960Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-10-10T01:23:14.4180372Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-10-10T01:23:14.4182429Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-10-10T01:23:14.4184199Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-10-10T01:23:14.4187141Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-10-10T01:23:14.4188896Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-10-10T01:23:14.4190990Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-10-10T01:23:14.4193504Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-10-10T01:23:14.4195194Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-10-10T01:23:14.4197146Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-10-10T01:23:14.4199630Z * [new branch] gh/etaf/162/base -> origin/gh/etaf/162/base 2025-10-10T01:23:14.4201383Z * [new branch] gh/etaf/162/head -> origin/gh/etaf/162/head 2025-10-10T01:23:14.4203228Z * [new branch] gh/etaf/162/orig -> origin/gh/etaf/162/orig 2025-10-10T01:23:14.4205517Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-10-10T01:23:14.4207299Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-10-10T01:23:14.4208983Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-10-10T01:23:14.4211233Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-10-10T01:23:14.4213144Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-10-10T01:23:14.4214838Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-10-10T01:23:14.4217212Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-10-10T01:23:14.4218962Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-10-10T01:23:14.4220673Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-10-10T01:23:14.4222980Z * [new branch] gh/etaf/170/base -> origin/gh/etaf/170/base 2025-10-10T01:23:14.4224739Z * [new branch] gh/etaf/170/head -> origin/gh/etaf/170/head 2025-10-10T01:23:14.4226624Z * [new branch] gh/etaf/170/orig -> origin/gh/etaf/170/orig 2025-10-10T01:23:14.4228942Z * [new branch] gh/etaf/171/base -> origin/gh/etaf/171/base 2025-10-10T01:23:14.4230567Z * [new branch] gh/etaf/171/head -> origin/gh/etaf/171/head 2025-10-10T01:23:14.4232311Z * [new branch] gh/etaf/171/orig -> origin/gh/etaf/171/orig 2025-10-10T01:23:14.4234505Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-10-10T01:23:14.4236213Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-10-10T01:23:14.4238432Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-10-10T01:23:14.4241802Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-10-10T01:23:14.4243185Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-10-10T01:23:14.4245710Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-10-10T01:23:14.4247052Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-10-10T01:23:14.4249807Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-10-10T01:23:14.4251013Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-10-10T01:23:14.4253613Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-10-10T01:23:14.4255182Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-10-10T01:23:14.4258340Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-10-10T01:23:14.4259955Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-10-10T01:23:14.4261706Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-10-10T01:23:14.4264064Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-10-10T01:23:14.4265738Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-10-10T01:23:14.4267450Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-10-10T01:23:14.4269883Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-10-10T01:23:14.4271217Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-10-10T01:23:14.4273109Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-10-10T01:23:14.4275357Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-10-10T01:23:14.4277146Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-10-10T01:23:14.4278799Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-10-10T01:23:14.4281220Z * [new branch] gh/ezyang/3122/base -> origin/gh/ezyang/3122/base 2025-10-10T01:23:14.4282901Z * [new branch] gh/ezyang/3122/head -> origin/gh/ezyang/3122/head 2025-10-10T01:23:14.4284658Z * [new branch] gh/ezyang/3122/orig -> origin/gh/ezyang/3122/orig 2025-10-10T01:23:14.4287423Z * [new branch] gh/ezyang/3127/base -> origin/gh/ezyang/3127/base 2025-10-10T01:23:14.4289086Z * [new branch] gh/ezyang/3127/head -> origin/gh/ezyang/3127/head 2025-10-10T01:23:14.4290760Z * [new branch] gh/ezyang/3127/orig -> origin/gh/ezyang/3127/orig 2025-10-10T01:23:14.4293309Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-10-10T01:23:14.4294978Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-10-10T01:23:14.4296819Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-10-10T01:23:14.4299120Z * [new branch] gh/ezyang/3134/base -> origin/gh/ezyang/3134/base 2025-10-10T01:23:14.4300802Z * [new branch] gh/ezyang/3134/head -> origin/gh/ezyang/3134/head 2025-10-10T01:23:14.4302573Z * [new branch] gh/ezyang/3134/orig -> origin/gh/ezyang/3134/orig 2025-10-10T01:23:14.4304862Z * [new branch] gh/ezyang/3135/base -> origin/gh/ezyang/3135/base 2025-10-10T01:23:14.4306511Z * [new branch] gh/ezyang/3135/head -> origin/gh/ezyang/3135/head 2025-10-10T01:23:14.4308242Z * [new branch] gh/ezyang/3135/orig -> origin/gh/ezyang/3135/orig 2025-10-10T01:23:14.4310546Z * [new branch] gh/ezyang/3138/base -> origin/gh/ezyang/3138/base 2025-10-10T01:23:14.4312168Z * [new branch] gh/ezyang/3138/head -> origin/gh/ezyang/3138/head 2025-10-10T01:23:14.4313916Z * [new branch] gh/ezyang/3138/orig -> origin/gh/ezyang/3138/orig 2025-10-10T01:23:14.4316292Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-10-10T01:23:14.4318106Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-10-10T01:23:14.4319766Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-10-10T01:23:14.4322046Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-10-10T01:23:14.4323706Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-10-10T01:23:14.4325443Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-10-10T01:23:14.4327677Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-10-10T01:23:14.4329403Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-10-10T01:23:14.4331070Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-10-10T01:23:14.4333470Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-10-10T01:23:14.4335127Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-10-10T01:23:14.4337372Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-10-10T01:23:14.4340227Z * [new branch] gh/ezyang/3145/base -> origin/gh/ezyang/3145/base 2025-10-10T01:23:14.4341780Z * [new branch] gh/ezyang/3145/head -> origin/gh/ezyang/3145/head 2025-10-10T01:23:14.4343556Z * [new branch] gh/ezyang/3145/orig -> origin/gh/ezyang/3145/orig 2025-10-10T01:23:14.4346378Z * [new branch] gh/ezyang/3146/base -> origin/gh/ezyang/3146/base 2025-10-10T01:23:14.4348084Z * [new branch] gh/ezyang/3146/head -> origin/gh/ezyang/3146/head 2025-10-10T01:23:14.4349784Z * [new branch] gh/ezyang/3146/orig -> origin/gh/ezyang/3146/orig 2025-10-10T01:23:14.4352057Z * [new branch] gh/ezyang/3147/base -> origin/gh/ezyang/3147/base 2025-10-10T01:23:14.4353750Z * [new branch] gh/ezyang/3147/head -> origin/gh/ezyang/3147/head 2025-10-10T01:23:14.4355411Z * [new branch] gh/ezyang/3147/orig -> origin/gh/ezyang/3147/orig 2025-10-10T01:23:14.4357768Z * [new branch] gh/ezyang/3148/base -> origin/gh/ezyang/3148/base 2025-10-10T01:23:14.4359536Z * [new branch] gh/ezyang/3148/head -> origin/gh/ezyang/3148/head 2025-10-10T01:23:14.4361292Z * [new branch] gh/ezyang/3148/orig -> origin/gh/ezyang/3148/orig 2025-10-10T01:23:14.4363760Z * [new branch] gh/ezyang/3149/base -> origin/gh/ezyang/3149/base 2025-10-10T01:23:14.4365446Z * [new branch] gh/ezyang/3149/head -> origin/gh/ezyang/3149/head 2025-10-10T01:23:14.4367150Z * [new branch] gh/ezyang/3149/orig -> origin/gh/ezyang/3149/orig 2025-10-10T01:23:14.4369524Z * [new branch] gh/ezyang/3150/base -> origin/gh/ezyang/3150/base 2025-10-10T01:23:14.4371289Z * [new branch] gh/ezyang/3150/head -> origin/gh/ezyang/3150/head 2025-10-10T01:23:14.4373417Z * [new branch] gh/ezyang/3150/orig -> origin/gh/ezyang/3150/orig 2025-10-10T01:23:14.4375839Z * [new branch] gh/ezyang/3151/base -> origin/gh/ezyang/3151/base 2025-10-10T01:23:14.4377544Z * [new branch] gh/ezyang/3151/head -> origin/gh/ezyang/3151/head 2025-10-10T01:23:14.4379306Z * [new branch] gh/ezyang/3151/orig -> origin/gh/ezyang/3151/orig 2025-10-10T01:23:14.4381578Z * [new branch] gh/ezyang/3152/base -> origin/gh/ezyang/3152/base 2025-10-10T01:23:14.4383235Z * [new branch] gh/ezyang/3152/head -> origin/gh/ezyang/3152/head 2025-10-10T01:23:14.4384947Z * [new branch] gh/ezyang/3152/orig -> origin/gh/ezyang/3152/orig 2025-10-10T01:23:14.4387439Z * [new branch] gh/ezyang/3153/base -> origin/gh/ezyang/3153/base 2025-10-10T01:23:14.4389014Z * [new branch] gh/ezyang/3153/head -> origin/gh/ezyang/3153/head 2025-10-10T01:23:14.4390777Z * [new branch] gh/ezyang/3153/orig -> origin/gh/ezyang/3153/orig 2025-10-10T01:23:14.4393243Z * [new branch] gh/ezyang/3154/base -> origin/gh/ezyang/3154/base 2025-10-10T01:23:14.4394895Z * [new branch] gh/ezyang/3154/head -> origin/gh/ezyang/3154/head 2025-10-10T01:23:14.4396881Z * [new branch] gh/ezyang/3154/orig -> origin/gh/ezyang/3154/orig 2025-10-10T01:23:14.4399525Z * [new branch] gh/ezyang/3155/base -> origin/gh/ezyang/3155/base 2025-10-10T01:23:14.4401275Z * [new branch] gh/ezyang/3155/head -> origin/gh/ezyang/3155/head 2025-10-10T01:23:14.4402928Z * [new branch] gh/ezyang/3155/orig -> origin/gh/ezyang/3155/orig 2025-10-10T01:23:14.4405319Z * [new branch] gh/ezyang/3156/base -> origin/gh/ezyang/3156/base 2025-10-10T01:23:14.4406971Z * [new branch] gh/ezyang/3156/head -> origin/gh/ezyang/3156/head 2025-10-10T01:23:14.4408803Z * [new branch] gh/ezyang/3156/orig -> origin/gh/ezyang/3156/orig 2025-10-10T01:23:14.4411181Z * [new branch] gh/ezyang/3157/base -> origin/gh/ezyang/3157/base 2025-10-10T01:23:14.4412836Z * [new branch] gh/ezyang/3157/head -> origin/gh/ezyang/3157/head 2025-10-10T01:23:14.4414588Z * [new branch] gh/ezyang/3157/orig -> origin/gh/ezyang/3157/orig 2025-10-10T01:23:14.4416968Z * [new branch] gh/ezyang/3158/base -> origin/gh/ezyang/3158/base 2025-10-10T01:23:14.4418675Z * [new branch] gh/ezyang/3158/head -> origin/gh/ezyang/3158/head 2025-10-10T01:23:14.4420306Z * [new branch] gh/ezyang/3158/orig -> origin/gh/ezyang/3158/orig 2025-10-10T01:23:14.4422849Z * [new branch] gh/ezyang/3159/base -> origin/gh/ezyang/3159/base 2025-10-10T01:23:14.4424569Z * [new branch] gh/ezyang/3159/head -> origin/gh/ezyang/3159/head 2025-10-10T01:23:14.4426232Z * [new branch] gh/ezyang/3159/orig -> origin/gh/ezyang/3159/orig 2025-10-10T01:23:14.4428643Z * [new branch] gh/ezyang/3160/base -> origin/gh/ezyang/3160/base 2025-10-10T01:23:14.4430328Z * [new branch] gh/ezyang/3160/head -> origin/gh/ezyang/3160/head 2025-10-10T01:23:14.4432040Z * [new branch] gh/ezyang/3160/orig -> origin/gh/ezyang/3160/orig 2025-10-10T01:23:14.4434500Z * [new branch] gh/ezyang/3161/base -> origin/gh/ezyang/3161/base 2025-10-10T01:23:14.4436166Z * [new branch] gh/ezyang/3161/head -> origin/gh/ezyang/3161/head 2025-10-10T01:23:14.4437869Z * [new branch] gh/ezyang/3161/orig -> origin/gh/ezyang/3161/orig 2025-10-10T01:23:14.4440339Z * [new branch] gh/ezyang/3162/base -> origin/gh/ezyang/3162/base 2025-10-10T01:23:14.4441984Z * [new branch] gh/ezyang/3162/head -> origin/gh/ezyang/3162/head 2025-10-10T01:23:14.4443694Z * [new branch] gh/ezyang/3162/orig -> origin/gh/ezyang/3162/orig 2025-10-10T01:23:14.4446048Z * [new branch] gh/ezyang/3163/base -> origin/gh/ezyang/3163/base 2025-10-10T01:23:14.4447762Z * [new branch] gh/ezyang/3163/head -> origin/gh/ezyang/3163/head 2025-10-10T01:23:14.4449497Z * [new branch] gh/ezyang/3163/orig -> origin/gh/ezyang/3163/orig 2025-10-10T01:23:14.4451844Z * [new branch] gh/ezyang/3164/base -> origin/gh/ezyang/3164/base 2025-10-10T01:23:14.4453568Z * [new branch] gh/ezyang/3164/head -> origin/gh/ezyang/3164/head 2025-10-10T01:23:14.4455283Z * [new branch] gh/ezyang/3164/orig -> origin/gh/ezyang/3164/orig 2025-10-10T01:23:14.4457829Z * [new branch] gh/ezyang/3165/base -> origin/gh/ezyang/3165/base 2025-10-10T01:23:14.4459514Z * [new branch] gh/ezyang/3165/head -> origin/gh/ezyang/3165/head 2025-10-10T01:23:14.4461757Z * [new branch] gh/ezyang/3165/orig -> origin/gh/ezyang/3165/orig 2025-10-10T01:23:14.4464134Z * [new branch] gh/ezyang/3166/base -> origin/gh/ezyang/3166/base 2025-10-10T01:23:14.4465805Z * [new branch] gh/ezyang/3166/head -> origin/gh/ezyang/3166/head 2025-10-10T01:23:14.4467491Z * [new branch] gh/ezyang/3166/orig -> origin/gh/ezyang/3166/orig 2025-10-10T01:23:14.4469884Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-10-10T01:23:14.4471621Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-10-10T01:23:14.4473329Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-10-10T01:23:14.4475757Z * [new branch] gh/ezyang/3168/base -> origin/gh/ezyang/3168/base 2025-10-10T01:23:14.4477458Z * [new branch] gh/ezyang/3168/head -> origin/gh/ezyang/3168/head 2025-10-10T01:23:14.4479253Z * [new branch] gh/ezyang/3168/orig -> origin/gh/ezyang/3168/orig 2025-10-10T01:23:14.4481695Z * [new branch] gh/ezyang/3169/base -> origin/gh/ezyang/3169/base 2025-10-10T01:23:14.4483343Z * [new branch] gh/ezyang/3169/head -> origin/gh/ezyang/3169/head 2025-10-10T01:23:14.4485061Z * [new branch] gh/ezyang/3169/orig -> origin/gh/ezyang/3169/orig 2025-10-10T01:23:14.4487455Z * [new branch] gh/ezyang/3170/base -> origin/gh/ezyang/3170/base 2025-10-10T01:23:14.4489134Z * [new branch] gh/ezyang/3170/head -> origin/gh/ezyang/3170/head 2025-10-10T01:23:14.4490966Z * [new branch] gh/ezyang/3170/orig -> origin/gh/ezyang/3170/orig 2025-10-10T01:23:14.4493315Z * [new branch] gh/ezyang/3171/base -> origin/gh/ezyang/3171/base 2025-10-10T01:23:14.4495024Z * [new branch] gh/ezyang/3171/head -> origin/gh/ezyang/3171/head 2025-10-10T01:23:14.4496800Z * [new branch] gh/ezyang/3171/orig -> origin/gh/ezyang/3171/orig 2025-10-10T01:23:14.4501010Z * [new branch] gh/ezyang/3172/base -> origin/gh/ezyang/3172/base 2025-10-10T01:23:14.4502667Z * [new branch] gh/ezyang/3172/head -> origin/gh/ezyang/3172/head 2025-10-10T01:23:14.4504442Z * [new branch] gh/ezyang/3172/orig -> origin/gh/ezyang/3172/orig 2025-10-10T01:23:14.4506886Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-10-10T01:23:14.4508551Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-10-10T01:23:14.4510202Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-10-10T01:23:14.4512862Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-10-10T01:23:14.4514537Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-10-10T01:23:14.4516234Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-10-10T01:23:14.4519710Z * [new branch] gh/fduwjj/175/base -> origin/gh/fduwjj/175/base 2025-10-10T01:23:14.4521607Z * [new branch] gh/fduwjj/175/head -> origin/gh/fduwjj/175/head 2025-10-10T01:23:14.4523335Z * [new branch] gh/fduwjj/175/orig -> origin/gh/fduwjj/175/orig 2025-10-10T01:23:14.4525754Z * [new branch] gh/fduwjj/176/base -> origin/gh/fduwjj/176/base 2025-10-10T01:23:14.4527499Z * [new branch] gh/fduwjj/176/head -> origin/gh/fduwjj/176/head 2025-10-10T01:23:14.4529337Z * [new branch] gh/fduwjj/176/orig -> origin/gh/fduwjj/176/orig 2025-10-10T01:23:14.4531540Z * [new branch] gh/fduwjj/177/base -> origin/gh/fduwjj/177/base 2025-10-10T01:23:14.4533349Z * [new branch] gh/fduwjj/177/head -> origin/gh/fduwjj/177/head 2025-10-10T01:23:14.4535012Z * [new branch] gh/fduwjj/177/orig -> origin/gh/fduwjj/177/orig 2025-10-10T01:23:14.4537309Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-10-10T01:23:14.4538992Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-10-10T01:23:14.4540693Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-10-10T01:23:14.4543103Z * [new branch] gh/fduwjj/183/base -> origin/gh/fduwjj/183/base 2025-10-10T01:23:14.4544904Z * [new branch] gh/fduwjj/183/head -> origin/gh/fduwjj/183/head 2025-10-10T01:23:14.4546657Z * [new branch] gh/fduwjj/183/orig -> origin/gh/fduwjj/183/orig 2025-10-10T01:23:14.4549208Z * [new branch] gh/fduwjj/184/base -> origin/gh/fduwjj/184/base 2025-10-10T01:23:14.4550861Z * [new branch] gh/fduwjj/184/head -> origin/gh/fduwjj/184/head 2025-10-10T01:23:14.4552668Z * [new branch] gh/fduwjj/184/orig -> origin/gh/fduwjj/184/orig 2025-10-10T01:23:14.4554969Z * [new branch] gh/fduwjj/185/base -> origin/gh/fduwjj/185/base 2025-10-10T01:23:14.4556688Z * [new branch] gh/fduwjj/185/head -> origin/gh/fduwjj/185/head 2025-10-10T01:23:14.4558447Z * [new branch] gh/fduwjj/185/orig -> origin/gh/fduwjj/185/orig 2025-10-10T01:23:14.4560946Z * [new branch] gh/fduwjj/191/base -> origin/gh/fduwjj/191/base 2025-10-10T01:23:14.4562686Z * [new branch] gh/fduwjj/191/head -> origin/gh/fduwjj/191/head 2025-10-10T01:23:14.4564421Z * [new branch] gh/fduwjj/191/orig -> origin/gh/fduwjj/191/orig 2025-10-10T01:23:14.4566741Z * [new branch] gh/fduwjj/192/base -> origin/gh/fduwjj/192/base 2025-10-10T01:23:14.4568527Z * [new branch] gh/fduwjj/192/head -> origin/gh/fduwjj/192/head 2025-10-10T01:23:14.4570180Z * [new branch] gh/fduwjj/192/orig -> origin/gh/fduwjj/192/orig 2025-10-10T01:23:14.4572596Z * [new branch] gh/fduwjj/193/base -> origin/gh/fduwjj/193/base 2025-10-10T01:23:14.4574269Z * [new branch] gh/fduwjj/193/head -> origin/gh/fduwjj/193/head 2025-10-10T01:23:14.4576016Z * [new branch] gh/fduwjj/193/orig -> origin/gh/fduwjj/193/orig 2025-10-10T01:23:14.4578443Z * [new branch] gh/fduwjj/194/base -> origin/gh/fduwjj/194/base 2025-10-10T01:23:14.4580170Z * [new branch] gh/fduwjj/194/head -> origin/gh/fduwjj/194/head 2025-10-10T01:23:14.4581880Z * [new branch] gh/fduwjj/194/orig -> origin/gh/fduwjj/194/orig 2025-10-10T01:23:14.4584313Z * [new branch] gh/fduwjj/195/base -> origin/gh/fduwjj/195/base 2025-10-10T01:23:14.4586196Z * [new branch] gh/fduwjj/195/head -> origin/gh/fduwjj/195/head 2025-10-10T01:23:14.4587892Z * [new branch] gh/fduwjj/195/orig -> origin/gh/fduwjj/195/orig 2025-10-10T01:23:14.4590065Z * [new branch] gh/fduwjj/196/base -> origin/gh/fduwjj/196/base 2025-10-10T01:23:14.4591755Z * [new branch] gh/fduwjj/196/head -> origin/gh/fduwjj/196/head 2025-10-10T01:23:14.4593437Z * [new branch] gh/fduwjj/196/orig -> origin/gh/fduwjj/196/orig 2025-10-10T01:23:14.4595611Z * [new branch] gh/fduwjj/197/base -> origin/gh/fduwjj/197/base 2025-10-10T01:23:14.4597631Z * [new branch] gh/fduwjj/197/head -> origin/gh/fduwjj/197/head 2025-10-10T01:23:14.4599598Z * [new branch] gh/fduwjj/197/orig -> origin/gh/fduwjj/197/orig 2025-10-10T01:23:14.4601735Z * [new branch] gh/fduwjj/198/base -> origin/gh/fduwjj/198/base 2025-10-10T01:23:14.4603481Z * [new branch] gh/fduwjj/198/head -> origin/gh/fduwjj/198/head 2025-10-10T01:23:14.4605196Z * [new branch] gh/fduwjj/198/orig -> origin/gh/fduwjj/198/orig 2025-10-10T01:23:14.4607334Z * [new branch] gh/fduwjj/199/base -> origin/gh/fduwjj/199/base 2025-10-10T01:23:14.4609057Z * [new branch] gh/fduwjj/199/head -> origin/gh/fduwjj/199/head 2025-10-10T01:23:14.4610717Z * [new branch] gh/fduwjj/199/orig -> origin/gh/fduwjj/199/orig 2025-10-10T01:23:14.4613344Z * [new branch] gh/fduwjj/200/base -> origin/gh/fduwjj/200/base 2025-10-10T01:23:14.4615016Z * [new branch] gh/fduwjj/200/head -> origin/gh/fduwjj/200/head 2025-10-10T01:23:14.4616704Z * [new branch] gh/fduwjj/200/orig -> origin/gh/fduwjj/200/orig 2025-10-10T01:23:14.4618967Z * [new branch] gh/fduwjj/201/base -> origin/gh/fduwjj/201/base 2025-10-10T01:23:14.4620710Z * [new branch] gh/fduwjj/201/head -> origin/gh/fduwjj/201/head 2025-10-10T01:23:14.4622387Z * [new branch] gh/fduwjj/201/orig -> origin/gh/fduwjj/201/orig 2025-10-10T01:23:14.4624840Z * [new branch] gh/fduwjj/202/base -> origin/gh/fduwjj/202/base 2025-10-10T01:23:14.4626511Z * [new branch] gh/fduwjj/202/head -> origin/gh/fduwjj/202/head 2025-10-10T01:23:14.4628191Z * [new branch] gh/fduwjj/202/orig -> origin/gh/fduwjj/202/orig 2025-10-10T01:23:14.4630794Z * [new branch] gh/fduwjj/203/base -> origin/gh/fduwjj/203/base 2025-10-10T01:23:14.4632634Z * [new branch] gh/fduwjj/203/head -> origin/gh/fduwjj/203/head 2025-10-10T01:23:14.4634319Z * [new branch] gh/fduwjj/203/orig -> origin/gh/fduwjj/203/orig 2025-10-10T01:23:14.4637417Z * [new branch] gh/fduwjj/204/base -> origin/gh/fduwjj/204/base 2025-10-10T01:23:14.4639204Z * [new branch] gh/fduwjj/204/head -> origin/gh/fduwjj/204/head 2025-10-10T01:23:14.4641028Z * [new branch] gh/fduwjj/204/orig -> origin/gh/fduwjj/204/orig 2025-10-10T01:23:14.4643557Z * [new branch] gh/fduwjj/205/base -> origin/gh/fduwjj/205/base 2025-10-10T01:23:14.4645251Z * [new branch] gh/fduwjj/205/head -> origin/gh/fduwjj/205/head 2025-10-10T01:23:14.4647115Z * [new branch] gh/fduwjj/205/orig -> origin/gh/fduwjj/205/orig 2025-10-10T01:23:14.4649537Z * [new branch] gh/fduwjj/206/base -> origin/gh/fduwjj/206/base 2025-10-10T01:23:14.4651355Z * [new branch] gh/fduwjj/206/head -> origin/gh/fduwjj/206/head 2025-10-10T01:23:14.4653022Z * [new branch] gh/fduwjj/206/orig -> origin/gh/fduwjj/206/orig 2025-10-10T01:23:14.4655569Z * [new branch] gh/fduwjj/207/base -> origin/gh/fduwjj/207/base 2025-10-10T01:23:14.4657208Z * [new branch] gh/fduwjj/207/head -> origin/gh/fduwjj/207/head 2025-10-10T01:23:14.4658930Z * [new branch] gh/fduwjj/207/orig -> origin/gh/fduwjj/207/orig 2025-10-10T01:23:14.4661195Z * [new branch] gh/fduwjj/208/base -> origin/gh/fduwjj/208/base 2025-10-10T01:23:14.4663036Z * [new branch] gh/fduwjj/208/head -> origin/gh/fduwjj/208/head 2025-10-10T01:23:14.4664785Z * [new branch] gh/fduwjj/208/orig -> origin/gh/fduwjj/208/orig 2025-10-10T01:23:14.4667043Z * [new branch] gh/fduwjj/209/base -> origin/gh/fduwjj/209/base 2025-10-10T01:23:14.4668861Z * [new branch] gh/fduwjj/209/head -> origin/gh/fduwjj/209/head 2025-10-10T01:23:14.4670465Z * [new branch] gh/fduwjj/209/orig -> origin/gh/fduwjj/209/orig 2025-10-10T01:23:14.4672680Z * [new branch] gh/fduwjj/210/base -> origin/gh/fduwjj/210/base 2025-10-10T01:23:14.4674380Z * [new branch] gh/fduwjj/210/head -> origin/gh/fduwjj/210/head 2025-10-10T01:23:14.4676156Z * [new branch] gh/fduwjj/210/orig -> origin/gh/fduwjj/210/orig 2025-10-10T01:23:14.4678559Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-10-10T01:23:14.4680417Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-10-10T01:23:14.4682193Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-10-10T01:23:14.4684513Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-10-10T01:23:14.4686193Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-10-10T01:23:14.4687875Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-10-10T01:23:14.4690268Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-10-10T01:23:14.4692044Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-10-10T01:23:14.4693936Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-10-10T01:23:14.4696513Z * [new branch] gh/fduwjj/214/base -> origin/gh/fduwjj/214/base 2025-10-10T01:23:14.4698366Z * [new branch] gh/fduwjj/214/head -> origin/gh/fduwjj/214/head 2025-10-10T01:23:14.4700036Z * [new branch] gh/fduwjj/214/orig -> origin/gh/fduwjj/214/orig 2025-10-10T01:23:14.4702863Z * [new branch] gh/fduwjj/215/base -> origin/gh/fduwjj/215/base 2025-10-10T01:23:14.4704640Z * [new branch] gh/fduwjj/215/head -> origin/gh/fduwjj/215/head 2025-10-10T01:23:14.4706363Z * [new branch] gh/fduwjj/215/orig -> origin/gh/fduwjj/215/orig 2025-10-10T01:23:14.4708807Z * [new branch] gh/fduwjj/216/base -> origin/gh/fduwjj/216/base 2025-10-10T01:23:14.4710444Z * [new branch] gh/fduwjj/216/head -> origin/gh/fduwjj/216/head 2025-10-10T01:23:14.4712044Z * [new branch] gh/fduwjj/216/orig -> origin/gh/fduwjj/216/orig 2025-10-10T01:23:14.4714644Z * [new branch] gh/fduwjj/217/base -> origin/gh/fduwjj/217/base 2025-10-10T01:23:14.4716231Z * [new branch] gh/fduwjj/217/head -> origin/gh/fduwjj/217/head 2025-10-10T01:23:14.4717993Z * [new branch] gh/fduwjj/217/orig -> origin/gh/fduwjj/217/orig 2025-10-10T01:23:14.4720531Z * [new branch] gh/fduwjj/218/base -> origin/gh/fduwjj/218/base 2025-10-10T01:23:14.4722231Z * [new branch] gh/fduwjj/218/head -> origin/gh/fduwjj/218/head 2025-10-10T01:23:14.4723879Z * [new branch] gh/fduwjj/218/orig -> origin/gh/fduwjj/218/orig 2025-10-10T01:23:14.4726336Z * [new branch] gh/fduwjj/219/base -> origin/gh/fduwjj/219/base 2025-10-10T01:23:14.4728124Z * [new branch] gh/fduwjj/219/head -> origin/gh/fduwjj/219/head 2025-10-10T01:23:14.4729800Z * [new branch] gh/fduwjj/219/orig -> origin/gh/fduwjj/219/orig 2025-10-10T01:23:14.4732160Z * [new branch] gh/fduwjj/220/base -> origin/gh/fduwjj/220/base 2025-10-10T01:23:14.4733958Z * [new branch] gh/fduwjj/220/head -> origin/gh/fduwjj/220/head 2025-10-10T01:23:14.4735692Z * [new branch] gh/fduwjj/220/orig -> origin/gh/fduwjj/220/orig 2025-10-10T01:23:14.4738056Z * [new branch] gh/fduwjj/221/base -> origin/gh/fduwjj/221/base 2025-10-10T01:23:14.4739934Z * [new branch] gh/fduwjj/221/head -> origin/gh/fduwjj/221/head 2025-10-10T01:23:14.4741610Z * [new branch] gh/fduwjj/221/orig -> origin/gh/fduwjj/221/orig 2025-10-10T01:23:14.4744097Z * [new branch] gh/fduwjj/222/base -> origin/gh/fduwjj/222/base 2025-10-10T01:23:14.4745563Z * [new branch] gh/fduwjj/222/head -> origin/gh/fduwjj/222/head 2025-10-10T01:23:14.4747239Z * [new branch] gh/fduwjj/222/orig -> origin/gh/fduwjj/222/orig 2025-10-10T01:23:14.4749641Z * [new branch] gh/fduwjj/223/base -> origin/gh/fduwjj/223/base 2025-10-10T01:23:14.4751388Z * [new branch] gh/fduwjj/223/head -> origin/gh/fduwjj/223/head 2025-10-10T01:23:14.4753026Z * [new branch] gh/fduwjj/223/orig -> origin/gh/fduwjj/223/orig 2025-10-10T01:23:14.4755793Z * [new branch] gh/fegin/313/base -> origin/gh/fegin/313/base 2025-10-10T01:23:14.4757481Z * [new branch] gh/fegin/313/head -> origin/gh/fegin/313/head 2025-10-10T01:23:14.4759273Z * [new branch] gh/fegin/313/orig -> origin/gh/fegin/313/orig 2025-10-10T01:23:14.4761815Z * [new branch] gh/fegin/314/base -> origin/gh/fegin/314/base 2025-10-10T01:23:14.4763516Z * [new branch] gh/fegin/314/head -> origin/gh/fegin/314/head 2025-10-10T01:23:14.4765190Z * [new branch] gh/fegin/314/orig -> origin/gh/fegin/314/orig 2025-10-10T01:23:14.4767526Z * [new branch] gh/fegin/315/base -> origin/gh/fegin/315/base 2025-10-10T01:23:14.4769282Z * [new branch] gh/fegin/315/head -> origin/gh/fegin/315/head 2025-10-10T01:23:14.4770862Z * [new branch] gh/fegin/315/orig -> origin/gh/fegin/315/orig 2025-10-10T01:23:14.4773129Z * [new branch] gh/fegin/316/base -> origin/gh/fegin/316/base 2025-10-10T01:23:14.4774816Z * [new branch] gh/fegin/316/head -> origin/gh/fegin/316/head 2025-10-10T01:23:14.4776514Z * [new branch] gh/fegin/316/orig -> origin/gh/fegin/316/orig 2025-10-10T01:23:14.4778774Z * [new branch] gh/fegin/317/base -> origin/gh/fegin/317/base 2025-10-10T01:23:14.4780543Z * [new branch] gh/fegin/317/head -> origin/gh/fegin/317/head 2025-10-10T01:23:14.4782184Z * [new branch] gh/fegin/317/orig -> origin/gh/fegin/317/orig 2025-10-10T01:23:14.4784469Z * [new branch] gh/fegin/318/base -> origin/gh/fegin/318/base 2025-10-10T01:23:14.4786257Z * [new branch] gh/fegin/318/head -> origin/gh/fegin/318/head 2025-10-10T01:23:14.4788504Z * [new branch] gh/fegin/318/orig -> origin/gh/fegin/318/orig 2025-10-10T01:23:14.4790881Z * [new branch] gh/fegin/319/base -> origin/gh/fegin/319/base 2025-10-10T01:23:14.4792577Z * [new branch] gh/fegin/319/head -> origin/gh/fegin/319/head 2025-10-10T01:23:14.4794319Z * [new branch] gh/fegin/319/orig -> origin/gh/fegin/319/orig 2025-10-10T01:23:14.4796920Z * [new branch] gh/fegin/320/base -> origin/gh/fegin/320/base 2025-10-10T01:23:14.4800995Z * [new branch] gh/fegin/320/head -> origin/gh/fegin/320/head 2025-10-10T01:23:14.4802711Z * [new branch] gh/fegin/320/orig -> origin/gh/fegin/320/orig 2025-10-10T01:23:14.4804987Z * [new branch] gh/fegin/321/base -> origin/gh/fegin/321/base 2025-10-10T01:23:14.4806737Z * [new branch] gh/fegin/321/head -> origin/gh/fegin/321/head 2025-10-10T01:23:14.4808420Z * [new branch] gh/fegin/321/orig -> origin/gh/fegin/321/orig 2025-10-10T01:23:14.4810601Z * [new branch] gh/fegin/322/base -> origin/gh/fegin/322/base 2025-10-10T01:23:14.4812455Z * [new branch] gh/fegin/322/head -> origin/gh/fegin/322/head 2025-10-10T01:23:14.4814091Z * [new branch] gh/fegin/322/orig -> origin/gh/fegin/322/orig 2025-10-10T01:23:14.4816355Z * [new branch] gh/fegin/323/base -> origin/gh/fegin/323/base 2025-10-10T01:23:14.4818049Z * [new branch] gh/fegin/323/head -> origin/gh/fegin/323/head 2025-10-10T01:23:14.4820421Z * [new branch] gh/fegin/324/base -> origin/gh/fegin/324/base 2025-10-10T01:23:14.4822234Z * [new branch] gh/fegin/324/head -> origin/gh/fegin/324/head 2025-10-10T01:23:14.4823954Z * [new branch] gh/fegin/324/orig -> origin/gh/fegin/324/orig 2025-10-10T01:23:14.4826244Z * [new branch] gh/fegin/325/base -> origin/gh/fegin/325/base 2025-10-10T01:23:14.4828037Z * [new branch] gh/fegin/325/head -> origin/gh/fegin/325/head 2025-10-10T01:23:14.4829830Z * [new branch] gh/fegin/325/orig -> origin/gh/fegin/325/orig 2025-10-10T01:23:14.4832224Z * [new branch] gh/fegin/326/base -> origin/gh/fegin/326/base 2025-10-10T01:23:14.4833939Z * [new branch] gh/fegin/326/head -> origin/gh/fegin/326/head 2025-10-10T01:23:14.4835845Z * [new branch] gh/fegin/326/orig -> origin/gh/fegin/326/orig 2025-10-10T01:23:14.4838089Z * [new branch] gh/fegin/327/base -> origin/gh/fegin/327/base 2025-10-10T01:23:14.4839833Z * [new branch] gh/fegin/327/head -> origin/gh/fegin/327/head 2025-10-10T01:23:14.4841980Z * [new branch] gh/fegin/327/orig -> origin/gh/fegin/327/orig 2025-10-10T01:23:14.4844778Z * [new branch] gh/fffrog/133/base -> origin/gh/fffrog/133/base 2025-10-10T01:23:14.4846504Z * [new branch] gh/fffrog/133/head -> origin/gh/fffrog/133/head 2025-10-10T01:23:14.4848195Z * [new branch] gh/fffrog/133/orig -> origin/gh/fffrog/133/orig 2025-10-10T01:23:14.4850463Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-10-10T01:23:14.4852165Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-10-10T01:23:14.4853874Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-10-10T01:23:14.4856118Z * [new branch] gh/fffrog/147/base -> origin/gh/fffrog/147/base 2025-10-10T01:23:14.4857897Z * [new branch] gh/fffrog/147/head -> origin/gh/fffrog/147/head 2025-10-10T01:23:14.4859624Z * [new branch] gh/fffrog/147/orig -> origin/gh/fffrog/147/orig 2025-10-10T01:23:14.4861924Z * [new branch] gh/fffrog/149/base -> origin/gh/fffrog/149/base 2025-10-10T01:23:14.4863635Z * [new branch] gh/fffrog/149/head -> origin/gh/fffrog/149/head 2025-10-10T01:23:14.4865399Z * [new branch] gh/fffrog/149/orig -> origin/gh/fffrog/149/orig 2025-10-10T01:23:14.4867723Z * [new branch] gh/fffrog/150/base -> origin/gh/fffrog/150/base 2025-10-10T01:23:14.4869406Z * [new branch] gh/fffrog/150/head -> origin/gh/fffrog/150/head 2025-10-10T01:23:14.4871183Z * [new branch] gh/fffrog/150/orig -> origin/gh/fffrog/150/orig 2025-10-10T01:23:14.4873480Z * [new branch] gh/fffrog/153/base -> origin/gh/fffrog/153/base 2025-10-10T01:23:14.4875165Z * [new branch] gh/fffrog/153/head -> origin/gh/fffrog/153/head 2025-10-10T01:23:14.4876845Z * [new branch] gh/fffrog/153/orig -> origin/gh/fffrog/153/orig 2025-10-10T01:23:14.4879238Z * [new branch] gh/fffrog/154/base -> origin/gh/fffrog/154/base 2025-10-10T01:23:14.4881161Z * [new branch] gh/fffrog/154/head -> origin/gh/fffrog/154/head 2025-10-10T01:23:14.4882720Z * [new branch] gh/fffrog/154/orig -> origin/gh/fffrog/154/orig 2025-10-10T01:23:14.4884976Z * [new branch] gh/fffrog/155/base -> origin/gh/fffrog/155/base 2025-10-10T01:23:14.4886619Z * [new branch] gh/fffrog/155/head -> origin/gh/fffrog/155/head 2025-10-10T01:23:14.4888305Z * [new branch] gh/fffrog/155/orig -> origin/gh/fffrog/155/orig 2025-10-10T01:23:14.4890551Z * [new branch] gh/fffrog/156/base -> origin/gh/fffrog/156/base 2025-10-10T01:23:14.4892414Z * [new branch] gh/fffrog/156/head -> origin/gh/fffrog/156/head 2025-10-10T01:23:14.4894084Z * [new branch] gh/fffrog/156/orig -> origin/gh/fffrog/156/orig 2025-10-10T01:23:14.4896572Z * [new branch] gh/fffrog/157/base -> origin/gh/fffrog/157/base 2025-10-10T01:23:14.4898277Z * [new branch] gh/fffrog/157/head -> origin/gh/fffrog/157/head 2025-10-10T01:23:14.4899977Z * [new branch] gh/fffrog/157/orig -> origin/gh/fffrog/157/orig 2025-10-10T01:23:14.4902415Z * [new branch] gh/fffrog/158/base -> origin/gh/fffrog/158/base 2025-10-10T01:23:14.4904266Z * [new branch] gh/fffrog/158/head -> origin/gh/fffrog/158/head 2025-10-10T01:23:14.4905988Z * [new branch] gh/fffrog/158/orig -> origin/gh/fffrog/158/orig 2025-10-10T01:23:14.4908313Z * [new branch] gh/fffrog/159/base -> origin/gh/fffrog/159/base 2025-10-10T01:23:14.4909992Z * [new branch] gh/fffrog/159/head -> origin/gh/fffrog/159/head 2025-10-10T01:23:14.4911713Z * [new branch] gh/fffrog/159/orig -> origin/gh/fffrog/159/orig 2025-10-10T01:23:14.4914046Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-10-10T01:23:14.4915771Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-10-10T01:23:14.4918049Z * [new branch] gh/fffrog/161/base -> origin/gh/fffrog/161/base 2025-10-10T01:23:14.4919802Z * [new branch] gh/fffrog/161/head -> origin/gh/fffrog/161/head 2025-10-10T01:23:14.4921536Z * [new branch] gh/fffrog/161/orig -> origin/gh/fffrog/161/orig 2025-10-10T01:23:14.4923846Z * [new branch] gh/fffrog/162/base -> origin/gh/fffrog/162/base 2025-10-10T01:23:14.4925493Z * [new branch] gh/fffrog/162/head -> origin/gh/fffrog/162/head 2025-10-10T01:23:14.4927296Z * [new branch] gh/fffrog/162/orig -> origin/gh/fffrog/162/orig 2025-10-10T01:23:14.4929645Z * [new branch] gh/fffrog/163/base -> origin/gh/fffrog/163/base 2025-10-10T01:23:14.4931405Z * [new branch] gh/fffrog/163/head -> origin/gh/fffrog/163/head 2025-10-10T01:23:14.4933074Z * [new branch] gh/fffrog/163/orig -> origin/gh/fffrog/163/orig 2025-10-10T01:23:14.4935817Z * [new branch] gh/fffrog/164/base -> origin/gh/fffrog/164/base 2025-10-10T01:23:14.4937538Z * [new branch] gh/fffrog/164/head -> origin/gh/fffrog/164/head 2025-10-10T01:23:14.4939236Z * [new branch] gh/fffrog/164/orig -> origin/gh/fffrog/164/orig 2025-10-10T01:23:14.4941531Z * [new branch] gh/fffrog/165/base -> origin/gh/fffrog/165/base 2025-10-10T01:23:14.4943245Z * [new branch] gh/fffrog/165/head -> origin/gh/fffrog/165/head 2025-10-10T01:23:14.4944931Z * [new branch] gh/fffrog/165/orig -> origin/gh/fffrog/165/orig 2025-10-10T01:23:14.4947753Z * [new branch] gh/fffrog/166/base -> origin/gh/fffrog/166/base 2025-10-10T01:23:14.4949464Z * [new branch] gh/fffrog/166/head -> origin/gh/fffrog/166/head 2025-10-10T01:23:14.4951377Z * [new branch] gh/fffrog/166/orig -> origin/gh/fffrog/166/orig 2025-10-10T01:23:14.4953737Z * [new branch] gh/fffrog/167/base -> origin/gh/fffrog/167/base 2025-10-10T01:23:14.4955818Z * [new branch] gh/fffrog/167/head -> origin/gh/fffrog/167/head 2025-10-10T01:23:14.4957516Z * [new branch] gh/fffrog/167/orig -> origin/gh/fffrog/167/orig 2025-10-10T01:23:14.4959969Z * [new branch] gh/fffrog/168/base -> origin/gh/fffrog/168/base 2025-10-10T01:23:14.4961701Z * [new branch] gh/fffrog/168/head -> origin/gh/fffrog/168/head 2025-10-10T01:23:14.4963469Z * [new branch] gh/fffrog/168/orig -> origin/gh/fffrog/168/orig 2025-10-10T01:23:14.4965790Z * [new branch] gh/fffrog/169/base -> origin/gh/fffrog/169/base 2025-10-10T01:23:14.4967458Z * [new branch] gh/fffrog/169/head -> origin/gh/fffrog/169/head 2025-10-10T01:23:14.4969208Z * [new branch] gh/fffrog/169/orig -> origin/gh/fffrog/169/orig 2025-10-10T01:23:14.4971431Z * [new branch] gh/fffrog/170/base -> origin/gh/fffrog/170/base 2025-10-10T01:23:14.4973082Z * [new branch] gh/fffrog/170/head -> origin/gh/fffrog/170/head 2025-10-10T01:23:14.4974943Z * [new branch] gh/fffrog/170/orig -> origin/gh/fffrog/170/orig 2025-10-10T01:23:14.4977334Z * [new branch] gh/fffrog/171/base -> origin/gh/fffrog/171/base 2025-10-10T01:23:14.4978987Z * [new branch] gh/fffrog/171/head -> origin/gh/fffrog/171/head 2025-10-10T01:23:14.4980675Z * [new branch] gh/fffrog/171/orig -> origin/gh/fffrog/171/orig 2025-10-10T01:23:14.4983119Z * [new branch] gh/fffrog/172/base -> origin/gh/fffrog/172/base 2025-10-10T01:23:14.4984812Z * [new branch] gh/fffrog/172/head -> origin/gh/fffrog/172/head 2025-10-10T01:23:14.4986533Z * [new branch] gh/fffrog/172/orig -> origin/gh/fffrog/172/orig 2025-10-10T01:23:14.4988913Z * [new branch] gh/fffrog/173/base -> origin/gh/fffrog/173/base 2025-10-10T01:23:14.4990599Z * [new branch] gh/fffrog/173/head -> origin/gh/fffrog/173/head 2025-10-10T01:23:14.4992205Z * [new branch] gh/fffrog/173/orig -> origin/gh/fffrog/173/orig 2025-10-10T01:23:14.4994500Z * [new branch] gh/fffrog/174/base -> origin/gh/fffrog/174/base 2025-10-10T01:23:14.4996322Z * [new branch] gh/fffrog/174/head -> origin/gh/fffrog/174/head 2025-10-10T01:23:14.4999844Z * [new branch] gh/fffrog/174/orig -> origin/gh/fffrog/174/orig 2025-10-10T01:23:14.5002357Z * [new branch] gh/fffrog/175/base -> origin/gh/fffrog/175/base 2025-10-10T01:23:14.5004150Z * [new branch] gh/fffrog/175/head -> origin/gh/fffrog/175/head 2025-10-10T01:23:14.5005885Z * [new branch] gh/fffrog/175/orig -> origin/gh/fffrog/175/orig 2025-10-10T01:23:14.5008350Z * [new branch] gh/fffrog/176/base -> origin/gh/fffrog/176/base 2025-10-10T01:23:14.5010069Z * [new branch] gh/fffrog/176/head -> origin/gh/fffrog/176/head 2025-10-10T01:23:14.5011755Z * [new branch] gh/fffrog/176/orig -> origin/gh/fffrog/176/orig 2025-10-10T01:23:14.5015101Z * [new branch] gh/fxdawnn/1/base -> origin/gh/fxdawnn/1/base 2025-10-10T01:23:14.5017168Z * [new branch] gh/fxdawnn/1/head -> origin/gh/fxdawnn/1/head 2025-10-10T01:23:14.5018811Z * [new branch] gh/fxdawnn/1/orig -> origin/gh/fxdawnn/1/orig 2025-10-10T01:23:14.5020882Z * [new branch] gh/fxdawnn/2/base -> origin/gh/fxdawnn/2/base 2025-10-10T01:23:14.5022695Z * [new branch] gh/fxdawnn/2/head -> origin/gh/fxdawnn/2/head 2025-10-10T01:23:14.5024601Z * [new branch] gh/fxdawnn/2/orig -> origin/gh/fxdawnn/2/orig 2025-10-10T01:23:14.5026673Z * [new branch] gh/fxdawnn/3/base -> origin/gh/fxdawnn/3/base 2025-10-10T01:23:14.5028360Z * [new branch] gh/fxdawnn/3/head -> origin/gh/fxdawnn/3/head 2025-10-10T01:23:14.5029983Z * [new branch] gh/fxdawnn/3/orig -> origin/gh/fxdawnn/3/orig 2025-10-10T01:23:14.5032334Z * [new branch] gh/fxdawnn/4/base -> origin/gh/fxdawnn/4/base 2025-10-10T01:23:14.5034046Z * [new branch] gh/fxdawnn/4/orig -> origin/gh/fxdawnn/4/orig 2025-10-10T01:23:14.5036819Z * [new branch] gh/gmagogsfm/1/base -> origin/gh/gmagogsfm/1/base 2025-10-10T01:23:14.5038974Z * [new branch] gh/gmagogsfm/1/head -> origin/gh/gmagogsfm/1/head 2025-10-10T01:23:14.5040851Z * [new branch] gh/gmagogsfm/1/orig -> origin/gh/gmagogsfm/1/orig 2025-10-10T01:23:14.5043024Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-10-10T01:23:14.5044737Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-10-10T01:23:14.5046436Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-10-10T01:23:14.5048625Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-10-10T01:23:14.5050360Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-10-10T01:23:14.5052008Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-10-10T01:23:14.5054990Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-10-10T01:23:14.5056721Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-10-10T01:23:14.5058365Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-10-10T01:23:14.5060580Z * [new branch] gh/guangyey/135/base -> origin/gh/guangyey/135/base 2025-10-10T01:23:14.5062342Z * [new branch] gh/guangyey/135/head -> origin/gh/guangyey/135/head 2025-10-10T01:23:14.5064101Z * [new branch] gh/guangyey/135/orig -> origin/gh/guangyey/135/orig 2025-10-10T01:23:14.5066398Z * [new branch] gh/guangyey/139/base -> origin/gh/guangyey/139/base 2025-10-10T01:23:14.5068117Z * [new branch] gh/guangyey/139/head -> origin/gh/guangyey/139/head 2025-10-10T01:23:14.5069797Z * [new branch] gh/guangyey/139/orig -> origin/gh/guangyey/139/orig 2025-10-10T01:23:14.5072211Z * [new branch] gh/guangyey/140/base -> origin/gh/guangyey/140/base 2025-10-10T01:23:14.5073846Z * [new branch] gh/guangyey/140/head -> origin/gh/guangyey/140/head 2025-10-10T01:23:14.5075552Z * [new branch] gh/guangyey/140/orig -> origin/gh/guangyey/140/orig 2025-10-10T01:23:14.5077868Z * [new branch] gh/guangyey/142/base -> origin/gh/guangyey/142/base 2025-10-10T01:23:14.5079752Z * [new branch] gh/guangyey/142/head -> origin/gh/guangyey/142/head 2025-10-10T01:23:14.5081404Z * [new branch] gh/guangyey/142/orig -> origin/gh/guangyey/142/orig 2025-10-10T01:23:14.5083657Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-10-10T01:23:14.5085416Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-10-10T01:23:14.5087115Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-10-10T01:23:14.5089455Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-10-10T01:23:14.5091209Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-10-10T01:23:14.5093069Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-10-10T01:23:14.5095262Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-10-10T01:23:14.5097075Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-10-10T01:23:14.5098836Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-10-10T01:23:14.5101055Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-10-10T01:23:14.5102715Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-10-10T01:23:14.5104413Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-10-10T01:23:14.5106752Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-10-10T01:23:14.5108479Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-10-10T01:23:14.5110212Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-10-10T01:23:14.5112562Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-10-10T01:23:14.5114349Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-10-10T01:23:14.5115987Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-10-10T01:23:14.5118322Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-10-10T01:23:14.5120123Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-10-10T01:23:14.5121836Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-10-10T01:23:14.5124617Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-10-10T01:23:14.5126391Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-10-10T01:23:14.5128099Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-10-10T01:23:14.5130349Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-10-10T01:23:14.5132080Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-10-10T01:23:14.5133738Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-10-10T01:23:14.5136045Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-10-10T01:23:14.5138016Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-10-10T01:23:14.5139865Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-10-10T01:23:14.5142107Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-10-10T01:23:14.5143786Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-10-10T01:23:14.5145448Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-10-10T01:23:14.5148289Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-10-10T01:23:14.5150030Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-10-10T01:23:14.5151692Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-10-10T01:23:14.5154114Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-10-10T01:23:14.5155832Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-10-10T01:23:14.5157574Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-10-10T01:23:14.5160165Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-10-10T01:23:14.5161978Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-10-10T01:23:14.5163617Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-10-10T01:23:14.5165910Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-10-10T01:23:14.5167587Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-10-10T01:23:14.5169262Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-10-10T01:23:14.5171661Z * [new branch] gh/guangyey/194/base -> origin/gh/guangyey/194/base 2025-10-10T01:23:14.5173317Z * [new branch] gh/guangyey/194/head -> origin/gh/guangyey/194/head 2025-10-10T01:23:14.5175003Z * [new branch] gh/guangyey/194/orig -> origin/gh/guangyey/194/orig 2025-10-10T01:23:14.5177370Z * [new branch] gh/guangyey/195/base -> origin/gh/guangyey/195/base 2025-10-10T01:23:14.5179214Z * [new branch] gh/guangyey/195/head -> origin/gh/guangyey/195/head 2025-10-10T01:23:14.5180899Z * [new branch] gh/guangyey/195/orig -> origin/gh/guangyey/195/orig 2025-10-10T01:23:14.5183340Z * [new branch] gh/guangyey/201/base -> origin/gh/guangyey/201/base 2025-10-10T01:23:14.5185203Z * [new branch] gh/guangyey/201/head -> origin/gh/guangyey/201/head 2025-10-10T01:23:14.5186862Z * [new branch] gh/guangyey/201/orig -> origin/gh/guangyey/201/orig 2025-10-10T01:23:14.5189197Z * [new branch] gh/guangyey/202/base -> origin/gh/guangyey/202/base 2025-10-10T01:23:14.5190816Z * [new branch] gh/guangyey/202/head -> origin/gh/guangyey/202/head 2025-10-10T01:23:14.5192459Z * [new branch] gh/guangyey/202/orig -> origin/gh/guangyey/202/orig 2025-10-10T01:23:14.5194820Z * [new branch] gh/guangyey/203/base -> origin/gh/guangyey/203/base 2025-10-10T01:23:14.5196746Z * [new branch] gh/guangyey/203/head -> origin/gh/guangyey/203/head 2025-10-10T01:23:14.5198981Z * [new branch] gh/guangyey/203/orig -> origin/gh/guangyey/203/orig 2025-10-10T01:23:14.5201718Z * [new branch] gh/guangyey/205/base -> origin/gh/guangyey/205/base 2025-10-10T01:23:14.5203266Z * [new branch] gh/guangyey/205/head -> origin/gh/guangyey/205/head 2025-10-10T01:23:14.5204962Z * [new branch] gh/guangyey/205/orig -> origin/gh/guangyey/205/orig 2025-10-10T01:23:14.5207567Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-10-10T01:23:14.5209688Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-10-10T01:23:14.5211317Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-10-10T01:23:14.5213762Z * [new branch] gh/guangyey/209/base -> origin/gh/guangyey/209/base 2025-10-10T01:23:14.5215421Z * [new branch] gh/guangyey/209/head -> origin/gh/guangyey/209/head 2025-10-10T01:23:14.5217102Z * [new branch] gh/guangyey/209/orig -> origin/gh/guangyey/209/orig 2025-10-10T01:23:14.5219629Z * [new branch] gh/guangyey/210/base -> origin/gh/guangyey/210/base 2025-10-10T01:23:14.5221357Z * [new branch] gh/guangyey/210/head -> origin/gh/guangyey/210/head 2025-10-10T01:23:14.5223057Z * [new branch] gh/guangyey/210/orig -> origin/gh/guangyey/210/orig 2025-10-10T01:23:14.5225425Z * [new branch] gh/guangyey/211/base -> origin/gh/guangyey/211/base 2025-10-10T01:23:14.5227098Z * [new branch] gh/guangyey/211/head -> origin/gh/guangyey/211/head 2025-10-10T01:23:14.5228796Z * [new branch] gh/guangyey/211/orig -> origin/gh/guangyey/211/orig 2025-10-10T01:23:14.5231444Z * [new branch] gh/guangyey/89/base -> origin/gh/guangyey/89/base 2025-10-10T01:23:14.5232846Z * [new branch] gh/guangyey/89/head -> origin/gh/guangyey/89/head 2025-10-10T01:23:14.5234739Z * [new branch] gh/guangyey/89/orig -> origin/gh/guangyey/89/orig 2025-10-10T01:23:14.5237824Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-10-10T01:23:14.5239552Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-10-10T01:23:14.5241391Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-10-10T01:23:14.5243589Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-10-10T01:23:14.5245268Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-10-10T01:23:14.5247163Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-10-10T01:23:14.5249538Z * [new branch] gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base 2025-10-10T01:23:14.5251122Z * [new branch] gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head 2025-10-10T01:23:14.5252989Z * [new branch] gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig 2025-10-10T01:23:14.5255321Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-10-10T01:23:14.5257064Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-10-10T01:23:14.5258720Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-10-10T01:23:14.5261190Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-10-10T01:23:14.5262916Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-10-10T01:23:14.5264457Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-10-10T01:23:14.5266741Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-10-10T01:23:14.5268453Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-10-10T01:23:14.5270220Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-10-10T01:23:14.5272715Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-10-10T01:23:14.5274324Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-10-10T01:23:14.5276197Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-10-10T01:23:14.5278414Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-10-10T01:23:14.5280281Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-10-10T01:23:14.5281909Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-10-10T01:23:14.5284656Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-10-10T01:23:14.5286374Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-10-10T01:23:14.5288069Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-10-10T01:23:14.5290357Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-10-10T01:23:14.5292045Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-10-10T01:23:14.5293752Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-10-10T01:23:14.5296365Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-10-10T01:23:14.5298171Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-10-10T01:23:14.5299811Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-10-10T01:23:14.5302209Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-10-10T01:23:14.5303843Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-10-10T01:23:14.5305583Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-10-10T01:23:14.5307900Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-10-10T01:23:14.5309618Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-10-10T01:23:14.5311504Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-10-10T01:23:14.5313781Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-10-10T01:23:14.5315451Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-10-10T01:23:14.5317206Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-10-10T01:23:14.5320631Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-10-10T01:23:14.5322253Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-10-10T01:23:14.5324010Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-10-10T01:23:14.5326391Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-10-10T01:23:14.5328180Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-10-10T01:23:14.5329891Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-10-10T01:23:14.5332672Z * [new branch] gh/guilhermeleobas/237/base -> origin/gh/guilhermeleobas/237/base 2025-10-10T01:23:14.5334437Z * [new branch] gh/guilhermeleobas/237/head -> origin/gh/guilhermeleobas/237/head 2025-10-10T01:23:14.5336134Z * [new branch] gh/guilhermeleobas/237/orig -> origin/gh/guilhermeleobas/237/orig 2025-10-10T01:23:14.5338491Z * [new branch] gh/guilhermeleobas/239/base -> origin/gh/guilhermeleobas/239/base 2025-10-10T01:23:14.5340280Z * [new branch] gh/guilhermeleobas/239/head -> origin/gh/guilhermeleobas/239/head 2025-10-10T01:23:14.5342023Z * [new branch] gh/guilhermeleobas/239/orig -> origin/gh/guilhermeleobas/239/orig 2025-10-10T01:23:14.5344415Z * [new branch] gh/guilhermeleobas/246/base -> origin/gh/guilhermeleobas/246/base 2025-10-10T01:23:14.5346142Z * [new branch] gh/guilhermeleobas/246/head -> origin/gh/guilhermeleobas/246/head 2025-10-10T01:23:14.5347856Z * [new branch] gh/guilhermeleobas/246/orig -> origin/gh/guilhermeleobas/246/orig 2025-10-10T01:23:14.5350326Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-10-10T01:23:14.5351983Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-10-10T01:23:14.5353718Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-10-10T01:23:14.5356059Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-10-10T01:23:14.5357794Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-10-10T01:23:14.5359586Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-10-10T01:23:14.5362093Z * [new branch] gh/guilhermeleobas/249/base -> origin/gh/guilhermeleobas/249/base 2025-10-10T01:23:14.5363672Z * [new branch] gh/guilhermeleobas/249/head -> origin/gh/guilhermeleobas/249/head 2025-10-10T01:23:14.5365406Z * [new branch] gh/guilhermeleobas/249/orig -> origin/gh/guilhermeleobas/249/orig 2025-10-10T01:23:14.5367924Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-10-10T01:23:14.5369630Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-10-10T01:23:14.5371381Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-10-10T01:23:14.5374311Z * [new branch] gh/henrylhtsang/150/base -> origin/gh/henrylhtsang/150/base 2025-10-10T01:23:14.5376011Z * [new branch] gh/henrylhtsang/150/head -> origin/gh/henrylhtsang/150/head 2025-10-10T01:23:14.5377759Z * [new branch] gh/henrylhtsang/150/orig -> origin/gh/henrylhtsang/150/orig 2025-10-10T01:23:14.5380157Z * [new branch] gh/henrylhtsang/151/base -> origin/gh/henrylhtsang/151/base 2025-10-10T01:23:14.5381833Z * [new branch] gh/henrylhtsang/151/head -> origin/gh/henrylhtsang/151/head 2025-10-10T01:23:14.5383594Z * [new branch] gh/henrylhtsang/151/orig -> origin/gh/henrylhtsang/151/orig 2025-10-10T01:23:14.5385914Z * [new branch] gh/henrylhtsang/152/base -> origin/gh/henrylhtsang/152/base 2025-10-10T01:23:14.5387670Z * [new branch] gh/henrylhtsang/152/head -> origin/gh/henrylhtsang/152/head 2025-10-10T01:23:14.5389399Z * [new branch] gh/henrylhtsang/152/orig -> origin/gh/henrylhtsang/152/orig 2025-10-10T01:23:14.5391569Z * [new branch] gh/henrylhtsang/153/base -> origin/gh/henrylhtsang/153/base 2025-10-10T01:23:14.5393237Z * [new branch] gh/henrylhtsang/153/head -> origin/gh/henrylhtsang/153/head 2025-10-10T01:23:14.5395049Z * [new branch] gh/henrylhtsang/153/orig -> origin/gh/henrylhtsang/153/orig 2025-10-10T01:23:14.5400036Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-10-10T01:23:14.5402181Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-10-10T01:23:14.5404489Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-10-10T01:23:14.5406778Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-10-10T01:23:14.5409011Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-10-10T01:23:14.5411431Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-10-10T01:23:14.5414280Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-10-10T01:23:14.5416204Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-10-10T01:23:14.5419080Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-10-10T01:23:14.5420727Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-10-10T01:23:14.5422929Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-10-10T01:23:14.5424622Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-10-10T01:23:14.5426314Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-10-10T01:23:14.5428681Z * [new branch] gh/isuruf/147/base -> origin/gh/isuruf/147/base 2025-10-10T01:23:14.5430922Z * [new branch] gh/isuruf/147/head -> origin/gh/isuruf/147/head 2025-10-10T01:23:14.5432671Z * [new branch] gh/isuruf/147/orig -> origin/gh/isuruf/147/orig 2025-10-10T01:23:14.5435011Z * [new branch] gh/isuruf/148/base -> origin/gh/isuruf/148/base 2025-10-10T01:23:14.5436936Z * [new branch] gh/isuruf/148/head -> origin/gh/isuruf/148/head 2025-10-10T01:23:14.5438486Z * [new branch] gh/isuruf/148/orig -> origin/gh/isuruf/148/orig 2025-10-10T01:23:14.5440857Z * [new branch] gh/isuruf/149/base -> origin/gh/isuruf/149/base 2025-10-10T01:23:14.5442509Z * [new branch] gh/isuruf/149/head -> origin/gh/isuruf/149/head 2025-10-10T01:23:14.5444209Z * [new branch] gh/isuruf/149/orig -> origin/gh/isuruf/149/orig 2025-10-10T01:23:14.5446544Z * [new branch] gh/isuruf/150/base -> origin/gh/isuruf/150/base 2025-10-10T01:23:14.5448255Z * [new branch] gh/isuruf/150/head -> origin/gh/isuruf/150/head 2025-10-10T01:23:14.5449907Z * [new branch] gh/isuruf/150/orig -> origin/gh/isuruf/150/orig 2025-10-10T01:23:14.5452115Z * [new branch] gh/isuruf/151/base -> origin/gh/isuruf/151/base 2025-10-10T01:23:14.5453850Z * [new branch] gh/isuruf/151/head -> origin/gh/isuruf/151/head 2025-10-10T01:23:14.5455599Z * [new branch] gh/isuruf/151/orig -> origin/gh/isuruf/151/orig 2025-10-10T01:23:14.5463549Z * [new branch] gh/isuruf/152/base -> origin/gh/isuruf/152/base 2025-10-10T01:23:14.5463971Z * [new branch] gh/isuruf/152/head -> origin/gh/isuruf/152/head 2025-10-10T01:23:14.5464341Z * [new branch] gh/isuruf/152/orig -> origin/gh/isuruf/152/orig 2025-10-10T01:23:14.5464705Z * [new branch] gh/isuruf/153/base -> origin/gh/isuruf/153/base 2025-10-10T01:23:14.5465230Z * [new branch] gh/isuruf/153/head -> origin/gh/isuruf/153/head 2025-10-10T01:23:14.5467122Z * [new branch] gh/isuruf/153/orig -> origin/gh/isuruf/153/orig 2025-10-10T01:23:14.5469919Z * [new branch] gh/isuruf/154/base -> origin/gh/isuruf/154/base 2025-10-10T01:23:14.5471601Z * [new branch] gh/isuruf/154/head -> origin/gh/isuruf/154/head 2025-10-10T01:23:14.5473221Z * [new branch] gh/isuruf/154/orig -> origin/gh/isuruf/154/orig 2025-10-10T01:23:14.5475525Z * [new branch] gh/isuruf/155/base -> origin/gh/isuruf/155/base 2025-10-10T01:23:14.5477210Z * [new branch] gh/isuruf/155/head -> origin/gh/isuruf/155/head 2025-10-10T01:23:14.5478930Z * [new branch] gh/isuruf/155/orig -> origin/gh/isuruf/155/orig 2025-10-10T01:23:14.5481269Z * [new branch] gh/isuruf/156/base -> origin/gh/isuruf/156/base 2025-10-10T01:23:14.5483025Z * [new branch] gh/isuruf/156/head -> origin/gh/isuruf/156/head 2025-10-10T01:23:14.5484720Z * [new branch] gh/isuruf/156/orig -> origin/gh/isuruf/156/orig 2025-10-10T01:23:14.5487072Z * [new branch] gh/isuruf/157/base -> origin/gh/isuruf/157/base 2025-10-10T01:23:14.5488749Z * [new branch] gh/isuruf/157/head -> origin/gh/isuruf/157/head 2025-10-10T01:23:14.5490476Z * [new branch] gh/isuruf/157/orig -> origin/gh/isuruf/157/orig 2025-10-10T01:23:14.5492727Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-10-10T01:23:14.5494400Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-10-10T01:23:14.5496242Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-10-10T01:23:14.5499214Z * [new branch] gh/jamesjwu/171/base -> origin/gh/jamesjwu/171/base 2025-10-10T01:23:14.5500960Z * [new branch] gh/jamesjwu/171/head -> origin/gh/jamesjwu/171/head 2025-10-10T01:23:14.5503184Z * [new branch] gh/jamesjwu/171/orig -> origin/gh/jamesjwu/171/orig 2025-10-10T01:23:14.5505547Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-10-10T01:23:14.5507397Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-10-10T01:23:14.5509005Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-10-10T01:23:14.5511252Z * [new branch] gh/jamesjwu/186/base -> origin/gh/jamesjwu/186/base 2025-10-10T01:23:14.5512956Z * [new branch] gh/jamesjwu/186/head -> origin/gh/jamesjwu/186/head 2025-10-10T01:23:14.5514642Z * [new branch] gh/jamesjwu/186/orig -> origin/gh/jamesjwu/186/orig 2025-10-10T01:23:14.5516942Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-10-10T01:23:14.5518619Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-10-10T01:23:14.5520440Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-10-10T01:23:14.5523213Z * [new branch] gh/jamesjwu/189/base -> origin/gh/jamesjwu/189/base 2025-10-10T01:23:14.5524916Z * [new branch] gh/jamesjwu/189/head -> origin/gh/jamesjwu/189/head 2025-10-10T01:23:14.5526609Z * [new branch] gh/jamesjwu/189/orig -> origin/gh/jamesjwu/189/orig 2025-10-10T01:23:14.5528978Z * [new branch] gh/jamesjwu/190/base -> origin/gh/jamesjwu/190/base 2025-10-10T01:23:14.5530637Z * [new branch] gh/jamesjwu/190/head -> origin/gh/jamesjwu/190/head 2025-10-10T01:23:14.5532338Z * [new branch] gh/jamesjwu/190/orig -> origin/gh/jamesjwu/190/orig 2025-10-10T01:23:14.5534636Z * [new branch] gh/jamesjwu/191/base -> origin/gh/jamesjwu/191/base 2025-10-10T01:23:14.5536359Z * [new branch] gh/jamesjwu/191/head -> origin/gh/jamesjwu/191/head 2025-10-10T01:23:14.5538037Z * [new branch] gh/jamesjwu/191/orig -> origin/gh/jamesjwu/191/orig 2025-10-10T01:23:14.5540389Z * [new branch] gh/jamesjwu/192/base -> origin/gh/jamesjwu/192/base 2025-10-10T01:23:14.5542162Z * [new branch] gh/jamesjwu/192/head -> origin/gh/jamesjwu/192/head 2025-10-10T01:23:14.5544520Z * [new branch] gh/jamesjwu/193/base -> origin/gh/jamesjwu/193/base 2025-10-10T01:23:14.5546208Z * [new branch] gh/jamesjwu/193/head -> origin/gh/jamesjwu/193/head 2025-10-10T01:23:14.5547896Z * [new branch] gh/jamesjwu/193/orig -> origin/gh/jamesjwu/193/orig 2025-10-10T01:23:14.5550395Z * [new branch] gh/jamesjwu/194/base -> origin/gh/jamesjwu/194/base 2025-10-10T01:23:14.5552179Z * [new branch] gh/jamesjwu/194/head -> origin/gh/jamesjwu/194/head 2025-10-10T01:23:14.5553957Z * [new branch] gh/jamesjwu/194/orig -> origin/gh/jamesjwu/194/orig 2025-10-10T01:23:14.5556231Z * [new branch] gh/jamesjwu/195/base -> origin/gh/jamesjwu/195/base 2025-10-10T01:23:14.5557853Z * [new branch] gh/jamesjwu/195/head -> origin/gh/jamesjwu/195/head 2025-10-10T01:23:14.5559599Z * [new branch] gh/jamesjwu/195/orig -> origin/gh/jamesjwu/195/orig 2025-10-10T01:23:14.5561915Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-10-10T01:23:14.5563664Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-10-10T01:23:14.5565403Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-10-10T01:23:14.5567802Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-10-10T01:23:14.5569487Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-10-10T01:23:14.5571702Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-10-10T01:23:14.5573309Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-10-10T01:23:14.5575648Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-10-10T01:23:14.5577217Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-10-10T01:23:14.5579416Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-10-10T01:23:14.5581018Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-10-10T01:23:14.5583168Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-10-10T01:23:14.5584811Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-10-10T01:23:14.5587549Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-10-10T01:23:14.5589166Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-10-10T01:23:14.5591447Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-10-10T01:23:14.5593098Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-10-10T01:23:14.5595512Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-10-10T01:23:14.5597566Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-10-10T01:23:14.5599865Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-10-10T01:23:14.5601583Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-10-10T01:23:14.5603844Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-10-10T01:23:14.5605463Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-10-10T01:23:14.5607714Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-10-10T01:23:14.5609365Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-10-10T01:23:14.5611580Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-10-10T01:23:14.5613730Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-10-10T01:23:14.5616185Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-10-10T01:23:14.5618134Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-10-10T01:23:14.5620391Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-10-10T01:23:14.5622033Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-10-10T01:23:14.5624955Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-10-10T01:23:14.5626929Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-10-10T01:23:14.5628980Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-10-10T01:23:14.5631966Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-10-10T01:23:14.5634196Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-10-10T01:23:14.5635797Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-10-10T01:23:14.5638928Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-10-10T01:23:14.5640771Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-10-10T01:23:14.5642584Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-10-10T01:23:14.5644947Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-10-10T01:23:14.5646674Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-10-10T01:23:14.5648564Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-10-10T01:23:14.5651143Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-10-10T01:23:14.5652826Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-10-10T01:23:14.5655017Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-10-10T01:23:14.5656637Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-10-10T01:23:14.5658925Z * [new branch] gh/janeyx99/304/base -> origin/gh/janeyx99/304/base 2025-10-10T01:23:14.5660622Z * [new branch] gh/janeyx99/304/head -> origin/gh/janeyx99/304/head 2025-10-10T01:23:14.5663072Z * [new branch] gh/janeyx99/304/orig -> origin/gh/janeyx99/304/orig 2025-10-10T01:23:14.5664989Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-10-10T01:23:14.5666268Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-10-10T01:23:14.5668474Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-10-10T01:23:14.5670144Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-10-10T01:23:14.5672500Z * [new branch] gh/janeyx99/307/base -> origin/gh/janeyx99/307/base 2025-10-10T01:23:14.5674190Z * [new branch] gh/janeyx99/307/head -> origin/gh/janeyx99/307/head 2025-10-10T01:23:14.5675841Z * [new branch] gh/janeyx99/307/orig -> origin/gh/janeyx99/307/orig 2025-10-10T01:23:14.5678019Z * [new branch] gh/janeyx99/308/base -> origin/gh/janeyx99/308/base 2025-10-10T01:23:14.5679832Z * [new branch] gh/janeyx99/308/head -> origin/gh/janeyx99/308/head 2025-10-10T01:23:14.5681528Z * [new branch] gh/janeyx99/308/orig -> origin/gh/janeyx99/308/orig 2025-10-10T01:23:14.5683840Z * [new branch] gh/janeyx99/309/base -> origin/gh/janeyx99/309/base 2025-10-10T01:23:14.5685557Z * [new branch] gh/janeyx99/309/head -> origin/gh/janeyx99/309/head 2025-10-10T01:23:14.5687406Z * [new branch] gh/janeyx99/309/orig -> origin/gh/janeyx99/309/orig 2025-10-10T01:23:14.5689808Z * [new branch] gh/janeyx99/310/base -> origin/gh/janeyx99/310/base 2025-10-10T01:23:14.5691480Z * [new branch] gh/janeyx99/310/head -> origin/gh/janeyx99/310/head 2025-10-10T01:23:14.5693150Z * [new branch] gh/janeyx99/310/orig -> origin/gh/janeyx99/310/orig 2025-10-10T01:23:14.5695316Z * [new branch] gh/janeyx99/311/base -> origin/gh/janeyx99/311/base 2025-10-10T01:23:14.5697318Z * [new branch] gh/janeyx99/311/head -> origin/gh/janeyx99/311/head 2025-10-10T01:23:14.5698953Z * [new branch] gh/janeyx99/311/orig -> origin/gh/janeyx99/311/orig 2025-10-10T01:23:14.5701058Z * [new branch] gh/janeyx99/312/base -> origin/gh/janeyx99/312/base 2025-10-10T01:23:14.5702770Z * [new branch] gh/janeyx99/312/head -> origin/gh/janeyx99/312/head 2025-10-10T01:23:14.5704499Z * [new branch] gh/janeyx99/312/orig -> origin/gh/janeyx99/312/orig 2025-10-10T01:23:14.5706626Z * [new branch] gh/janeyx99/313/base -> origin/gh/janeyx99/313/base 2025-10-10T01:23:14.5708307Z * [new branch] gh/janeyx99/313/head -> origin/gh/janeyx99/313/head 2025-10-10T01:23:14.5710062Z * [new branch] gh/janeyx99/313/orig -> origin/gh/janeyx99/313/orig 2025-10-10T01:23:14.5712959Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-10-10T01:23:14.5714721Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-10-10T01:23:14.5716548Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-10-10T01:23:14.5718922Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-10-10T01:23:14.5720932Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-10-10T01:23:14.5722523Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-10-10T01:23:14.5725840Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-10-10T01:23:14.5727523Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-10-10T01:23:14.5729805Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-10-10T01:23:14.5731544Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-10-10T01:23:14.5733364Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-10-10T01:23:14.5735611Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-10-10T01:23:14.5737289Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-10-10T01:23:14.5738986Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-10-10T01:23:14.5741243Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-10-10T01:23:14.5743360Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-10-10T01:23:14.5745526Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-10-10T01:23:14.5747806Z * [new branch] gh/jansel/532/base -> origin/gh/jansel/532/base 2025-10-10T01:23:14.5749466Z * [new branch] gh/jansel/532/head -> origin/gh/jansel/532/head 2025-10-10T01:23:14.5751182Z * [new branch] gh/jansel/532/orig -> origin/gh/jansel/532/orig 2025-10-10T01:23:14.5753472Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-10-10T01:23:14.5755145Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-10-10T01:23:14.5756920Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-10-10T01:23:14.5759260Z * [new branch] gh/jansel/534/base -> origin/gh/jansel/534/base 2025-10-10T01:23:14.5761020Z * [new branch] gh/jansel/534/head -> origin/gh/jansel/534/head 2025-10-10T01:23:14.5763109Z * [new branch] gh/jansel/534/orig -> origin/gh/jansel/534/orig 2025-10-10T01:23:14.5765351Z * [new branch] gh/jansel/535/base -> origin/gh/jansel/535/base 2025-10-10T01:23:14.5767071Z * [new branch] gh/jansel/535/head -> origin/gh/jansel/535/head 2025-10-10T01:23:14.5768755Z * [new branch] gh/jansel/535/orig -> origin/gh/jansel/535/orig 2025-10-10T01:23:14.5770995Z * [new branch] gh/jansel/536/base -> origin/gh/jansel/536/base 2025-10-10T01:23:14.5772689Z * [new branch] gh/jansel/536/head -> origin/gh/jansel/536/head 2025-10-10T01:23:14.5774547Z * [new branch] gh/jansel/536/orig -> origin/gh/jansel/536/orig 2025-10-10T01:23:14.5776669Z * [new branch] gh/jansel/537/base -> origin/gh/jansel/537/base 2025-10-10T01:23:14.5778315Z * [new branch] gh/jansel/537/head -> origin/gh/jansel/537/head 2025-10-10T01:23:14.5780165Z * [new branch] gh/jansel/537/orig -> origin/gh/jansel/537/orig 2025-10-10T01:23:14.5782438Z * [new branch] gh/jansel/538/base -> origin/gh/jansel/538/base 2025-10-10T01:23:14.5784138Z * [new branch] gh/jansel/538/head -> origin/gh/jansel/538/head 2025-10-10T01:23:14.5785941Z * [new branch] gh/jansel/538/orig -> origin/gh/jansel/538/orig 2025-10-10T01:23:14.5788198Z * [new branch] gh/jansel/539/base -> origin/gh/jansel/539/base 2025-10-10T01:23:14.5789950Z * [new branch] gh/jansel/539/head -> origin/gh/jansel/539/head 2025-10-10T01:23:14.5791987Z * [new branch] gh/jansel/539/orig -> origin/gh/jansel/539/orig 2025-10-10T01:23:14.5794301Z * [new branch] gh/jansel/540/base -> origin/gh/jansel/540/base 2025-10-10T01:23:14.5796213Z * [new branch] gh/jansel/540/head -> origin/gh/jansel/540/head 2025-10-10T01:23:14.5800592Z * [new branch] gh/jansel/540/orig -> origin/gh/jansel/540/orig 2025-10-10T01:23:14.5802960Z * [new branch] gh/jansel/541/base -> origin/gh/jansel/541/base 2025-10-10T01:23:14.5804658Z * [new branch] gh/jansel/541/head -> origin/gh/jansel/541/head 2025-10-10T01:23:14.5806418Z * [new branch] gh/jansel/541/orig -> origin/gh/jansel/541/orig 2025-10-10T01:23:14.5808765Z * [new branch] gh/jansel/542/base -> origin/gh/jansel/542/base 2025-10-10T01:23:14.5810460Z * [new branch] gh/jansel/542/head -> origin/gh/jansel/542/head 2025-10-10T01:23:14.5812146Z * [new branch] gh/jansel/542/orig -> origin/gh/jansel/542/orig 2025-10-10T01:23:14.5814389Z * [new branch] gh/jansel/543/base -> origin/gh/jansel/543/base 2025-10-10T01:23:14.5816096Z * [new branch] gh/jansel/543/head -> origin/gh/jansel/543/head 2025-10-10T01:23:14.5817787Z * [new branch] gh/jansel/543/orig -> origin/gh/jansel/543/orig 2025-10-10T01:23:14.5820495Z * [new branch] gh/jansel/544/base -> origin/gh/jansel/544/base 2025-10-10T01:23:14.5822278Z * [new branch] gh/jansel/544/head -> origin/gh/jansel/544/head 2025-10-10T01:23:14.5824021Z * [new branch] gh/jansel/544/orig -> origin/gh/jansel/544/orig 2025-10-10T01:23:14.5826522Z * [new branch] gh/jansel/545/base -> origin/gh/jansel/545/base 2025-10-10T01:23:14.5828195Z * [new branch] gh/jansel/545/head -> origin/gh/jansel/545/head 2025-10-10T01:23:14.5830030Z * [new branch] gh/jansel/545/orig -> origin/gh/jansel/545/orig 2025-10-10T01:23:14.5832406Z * [new branch] gh/jansel/546/base -> origin/gh/jansel/546/base 2025-10-10T01:23:14.5834077Z * [new branch] gh/jansel/546/head -> origin/gh/jansel/546/head 2025-10-10T01:23:14.5835756Z * [new branch] gh/jansel/546/orig -> origin/gh/jansel/546/orig 2025-10-10T01:23:14.5838180Z * [new branch] gh/jansel/547/base -> origin/gh/jansel/547/base 2025-10-10T01:23:14.5840136Z * [new branch] gh/jansel/547/head -> origin/gh/jansel/547/head 2025-10-10T01:23:14.5841815Z * [new branch] gh/jansel/547/orig -> origin/gh/jansel/547/orig 2025-10-10T01:23:14.5844101Z * [new branch] gh/jansel/548/base -> origin/gh/jansel/548/base 2025-10-10T01:23:14.5845808Z * [new branch] gh/jansel/548/head -> origin/gh/jansel/548/head 2025-10-10T01:23:14.5847540Z * [new branch] gh/jansel/548/orig -> origin/gh/jansel/548/orig 2025-10-10T01:23:14.5850347Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-10-10T01:23:14.5852108Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-10-10T01:23:14.5853819Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-10-10T01:23:14.5856170Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-10-10T01:23:14.5857874Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-10-10T01:23:14.5859730Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-10-10T01:23:14.5862311Z * [new branch] gh/jbschlosser/251/base -> origin/gh/jbschlosser/251/base 2025-10-10T01:23:14.5864051Z * [new branch] gh/jbschlosser/251/head -> origin/gh/jbschlosser/251/head 2025-10-10T01:23:14.5865775Z * [new branch] gh/jbschlosser/251/orig -> origin/gh/jbschlosser/251/orig 2025-10-10T01:23:14.5868599Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-10-10T01:23:14.5870421Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-10-10T01:23:14.5872437Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-10-10T01:23:14.5874805Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-10-10T01:23:14.5876481Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-10-10T01:23:14.5878187Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-10-10T01:23:14.5880624Z * [new branch] gh/jiayisunx/65/base -> origin/gh/jiayisunx/65/base 2025-10-10T01:23:14.5882310Z * [new branch] gh/jiayisunx/65/head -> origin/gh/jiayisunx/65/head 2025-10-10T01:23:14.5883974Z * [new branch] gh/jiayisunx/65/orig -> origin/gh/jiayisunx/65/orig 2025-10-10T01:23:14.5886311Z * [new branch] gh/jiayisunx/67/base -> origin/gh/jiayisunx/67/base 2025-10-10T01:23:14.5887975Z * [new branch] gh/jiayisunx/67/head -> origin/gh/jiayisunx/67/head 2025-10-10T01:23:14.5889674Z * [new branch] gh/jiayisunx/67/orig -> origin/gh/jiayisunx/67/orig 2025-10-10T01:23:14.5892019Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-10-10T01:23:14.5893702Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-10-10T01:23:14.5895525Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-10-10T01:23:14.5898142Z * [new branch] gh/jiayisunx/71/base -> origin/gh/jiayisunx/71/base 2025-10-10T01:23:14.5899835Z * [new branch] gh/jiayisunx/71/head -> origin/gh/jiayisunx/71/head 2025-10-10T01:23:14.5901512Z * [new branch] gh/jiayisunx/71/orig -> origin/gh/jiayisunx/71/orig 2025-10-10T01:23:14.5904345Z * [new branch] gh/jiayisunx/72/base -> origin/gh/jiayisunx/72/base 2025-10-10T01:23:14.5906058Z * [new branch] gh/jiayisunx/72/head -> origin/gh/jiayisunx/72/head 2025-10-10T01:23:14.5907894Z * [new branch] gh/jiayisunx/72/orig -> origin/gh/jiayisunx/72/orig 2025-10-10T01:23:14.5910243Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-10-10T01:23:14.5911982Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-10-10T01:23:14.5913683Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-10-10T01:23:14.5915922Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-10-10T01:23:14.5917646Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-10-10T01:23:14.5919391Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-10-10T01:23:14.5922314Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-10-10T01:23:14.5923955Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-10-10T01:23:14.5925698Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-10-10T01:23:14.5928002Z * [new branch] gh/jiayisunx/80/base -> origin/gh/jiayisunx/80/base 2025-10-10T01:23:14.5929822Z * [new branch] gh/jiayisunx/80/head -> origin/gh/jiayisunx/80/head 2025-10-10T01:23:14.5931473Z * [new branch] gh/jiayisunx/80/orig -> origin/gh/jiayisunx/80/orig 2025-10-10T01:23:14.5933790Z * [new branch] gh/jiayisunx/81/base -> origin/gh/jiayisunx/81/base 2025-10-10T01:23:14.5935482Z * [new branch] gh/jiayisunx/81/head -> origin/gh/jiayisunx/81/head 2025-10-10T01:23:14.5937227Z * [new branch] gh/jiayisunx/81/orig -> origin/gh/jiayisunx/81/orig 2025-10-10T01:23:14.5939506Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-10-10T01:23:14.5941396Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-10-10T01:23:14.5942989Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-10-10T01:23:14.5945556Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-10-10T01:23:14.5947627Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-10-10T01:23:14.5950148Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-10-10T01:23:14.5953296Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-10-10T01:23:14.5955678Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-10-10T01:23:14.5957975Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-10-10T01:23:14.5961698Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-10-10T01:23:14.5963828Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-10-10T01:23:14.5967600Z * [new branch] gh/karthickai/3/base -> origin/gh/karthickai/3/base 2025-10-10T01:23:14.5970087Z * [new branch] gh/karthickai/3/head -> origin/gh/karthickai/3/head 2025-10-10T01:23:14.5971970Z * [new branch] gh/karthickai/3/orig -> origin/gh/karthickai/3/orig 2025-10-10T01:23:14.5974456Z * [new branch] gh/karthickai/4/base -> origin/gh/karthickai/4/base 2025-10-10T01:23:14.5976251Z * [new branch] gh/karthickai/4/head -> origin/gh/karthickai/4/head 2025-10-10T01:23:14.5977978Z * [new branch] gh/karthickai/4/orig -> origin/gh/karthickai/4/orig 2025-10-10T01:23:14.5980335Z * [new branch] gh/karthickai/5/base -> origin/gh/karthickai/5/base 2025-10-10T01:23:14.5982097Z * [new branch] gh/karthickai/5/head -> origin/gh/karthickai/5/head 2025-10-10T01:23:14.5983816Z * [new branch] gh/karthickai/5/orig -> origin/gh/karthickai/5/orig 2025-10-10T01:23:14.5986183Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-10-10T01:23:14.5987928Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-10-10T01:23:14.5989706Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-10-10T01:23:14.5993058Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-10-10T01:23:14.5994784Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-10-10T01:23:14.5996856Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-10-10T01:23:14.5999180Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-10-10T01:23:14.6000984Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-10-10T01:23:14.6002669Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-10-10T01:23:14.6005045Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-10-10T01:23:14.6006928Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-10-10T01:23:14.6008542Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-10-10T01:23:14.6010854Z * [new branch] gh/kurtamohler/51/base -> origin/gh/kurtamohler/51/base 2025-10-10T01:23:14.6012531Z * [new branch] gh/kurtamohler/51/head -> origin/gh/kurtamohler/51/head 2025-10-10T01:23:14.6014231Z * [new branch] gh/kurtamohler/51/orig -> origin/gh/kurtamohler/51/orig 2025-10-10T01:23:14.6016574Z * [new branch] gh/kurtamohler/52/base -> origin/gh/kurtamohler/52/base 2025-10-10T01:23:14.6018334Z * [new branch] gh/kurtamohler/52/head -> origin/gh/kurtamohler/52/head 2025-10-10T01:23:14.6020103Z * [new branch] gh/kurtamohler/52/orig -> origin/gh/kurtamohler/52/orig 2025-10-10T01:23:14.6022519Z * [new branch] gh/kurtamohler/53/base -> origin/gh/kurtamohler/53/base 2025-10-10T01:23:14.6024242Z * [new branch] gh/kurtamohler/53/head -> origin/gh/kurtamohler/53/head 2025-10-10T01:23:14.6025888Z * [new branch] gh/kurtamohler/53/orig -> origin/gh/kurtamohler/53/orig 2025-10-10T01:23:14.6028179Z * [new branch] gh/kurtamohler/54/base -> origin/gh/kurtamohler/54/base 2025-10-10T01:23:14.6029846Z * [new branch] gh/kurtamohler/54/head -> origin/gh/kurtamohler/54/head 2025-10-10T01:23:14.6032082Z * [new branch] gh/kurtamohler/54/orig -> origin/gh/kurtamohler/54/orig 2025-10-10T01:23:14.6034457Z * [new branch] gh/kurtamohler/55/base -> origin/gh/kurtamohler/55/base 2025-10-10T01:23:14.6036297Z * [new branch] gh/kurtamohler/55/head -> origin/gh/kurtamohler/55/head 2025-10-10T01:23:14.6037927Z * [new branch] gh/kurtamohler/55/orig -> origin/gh/kurtamohler/55/orig 2025-10-10T01:23:14.6041014Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-10-10T01:23:14.6042834Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-10-10T01:23:14.6044630Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-10-10T01:23:14.6046923Z * [new branch] gh/kwen2501/15/base -> origin/gh/kwen2501/15/base 2025-10-10T01:23:14.6048628Z * [new branch] gh/kwen2501/15/head -> origin/gh/kwen2501/15/head 2025-10-10T01:23:14.6050888Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-10-10T01:23:14.6052566Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-10-10T01:23:14.6054958Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-10-10T01:23:14.6056738Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-10-10T01:23:14.6058426Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-10-10T01:23:14.6060738Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-10-10T01:23:14.6062407Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-10-10T01:23:14.6064098Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-10-10T01:23:14.6066425Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-10-10T01:23:14.6068140Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-10-10T01:23:14.6070451Z * [new branch] gh/kwen2501/222/base -> origin/gh/kwen2501/222/base 2025-10-10T01:23:14.6072120Z * [new branch] gh/kwen2501/222/head -> origin/gh/kwen2501/222/head 2025-10-10T01:23:14.6074385Z * [new branch] gh/kwen2501/222/orig -> origin/gh/kwen2501/222/orig 2025-10-10T01:23:14.6076643Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-10-10T01:23:14.6078364Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-10-10T01:23:14.6080176Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-10-10T01:23:14.6082423Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-10-10T01:23:14.6084105Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-10-10T01:23:14.6085814Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-10-10T01:23:14.6088286Z * [new branch] gh/kwen2501/230/base -> origin/gh/kwen2501/230/base 2025-10-10T01:23:14.6089970Z * [new branch] gh/kwen2501/230/head -> origin/gh/kwen2501/230/head 2025-10-10T01:23:14.6091724Z * [new branch] gh/kwen2501/230/orig -> origin/gh/kwen2501/230/orig 2025-10-10T01:23:14.6094044Z * [new branch] gh/kwen2501/231/base -> origin/gh/kwen2501/231/base 2025-10-10T01:23:14.6095824Z * [new branch] gh/kwen2501/231/head -> origin/gh/kwen2501/231/head 2025-10-10T01:23:14.6097803Z * [new branch] gh/kwen2501/231/orig -> origin/gh/kwen2501/231/orig 2025-10-10T01:23:14.6100043Z * [new branch] gh/kwen2501/232/base -> origin/gh/kwen2501/232/base 2025-10-10T01:23:14.6101754Z * [new branch] gh/kwen2501/232/head -> origin/gh/kwen2501/232/head 2025-10-10T01:23:14.6103454Z * [new branch] gh/kwen2501/232/orig -> origin/gh/kwen2501/232/orig 2025-10-10T01:23:14.6105720Z * [new branch] gh/kwen2501/233/base -> origin/gh/kwen2501/233/base 2025-10-10T01:23:14.6107375Z * [new branch] gh/kwen2501/233/head -> origin/gh/kwen2501/233/head 2025-10-10T01:23:14.6109072Z * [new branch] gh/kwen2501/233/orig -> origin/gh/kwen2501/233/orig 2025-10-10T01:23:14.6111582Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-10-10T01:23:14.6113302Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-10-10T01:23:14.6114958Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-10-10T01:23:14.6117252Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-10-10T01:23:14.6118937Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-10-10T01:23:14.6120726Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-10-10T01:23:14.6122890Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-10-10T01:23:14.6124610Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-10-10T01:23:14.6126315Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-10-10T01:23:14.6128569Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-10-10T01:23:14.6130378Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-10-10T01:23:14.6132154Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-10-10T01:23:14.6134964Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-10-10T01:23:14.6136666Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-10-10T01:23:14.6138414Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-10-10T01:23:14.6140806Z * [new branch] gh/kwen2501/239/base -> origin/gh/kwen2501/239/base 2025-10-10T01:23:14.6142642Z * [new branch] gh/kwen2501/239/head -> origin/gh/kwen2501/239/head 2025-10-10T01:23:14.6144211Z * [new branch] gh/kwen2501/239/orig -> origin/gh/kwen2501/239/orig 2025-10-10T01:23:14.6147100Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-10-10T01:23:14.6148816Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-10-10T01:23:14.6150470Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-10-10T01:23:14.6152801Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-10-10T01:23:14.6154500Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-10-10T01:23:14.6156199Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-10-10T01:23:14.6158521Z * [new branch] gh/kwen2501/242/base -> origin/gh/kwen2501/242/base 2025-10-10T01:23:14.6160381Z * [new branch] gh/kwen2501/242/head -> origin/gh/kwen2501/242/head 2025-10-10T01:23:14.6162080Z * [new branch] gh/kwen2501/242/orig -> origin/gh/kwen2501/242/orig 2025-10-10T01:23:14.6164349Z * [new branch] gh/kwen2501/243/base -> origin/gh/kwen2501/243/base 2025-10-10T01:23:14.6166102Z * [new branch] gh/kwen2501/243/head -> origin/gh/kwen2501/243/head 2025-10-10T01:23:14.6167827Z * [new branch] gh/kwen2501/243/orig -> origin/gh/kwen2501/243/orig 2025-10-10T01:23:14.6170086Z * [new branch] gh/kwen2501/244/base -> origin/gh/kwen2501/244/base 2025-10-10T01:23:14.6171808Z * [new branch] gh/kwen2501/244/head -> origin/gh/kwen2501/244/head 2025-10-10T01:23:14.6173491Z * [new branch] gh/kwen2501/244/orig -> origin/gh/kwen2501/244/orig 2025-10-10T01:23:14.6175764Z * [new branch] gh/kwen2501/245/base -> origin/gh/kwen2501/245/base 2025-10-10T01:23:14.6177507Z * [new branch] gh/kwen2501/245/head -> origin/gh/kwen2501/245/head 2025-10-10T01:23:14.6179208Z * [new branch] gh/kwen2501/245/orig -> origin/gh/kwen2501/245/orig 2025-10-10T01:23:14.6181474Z * [new branch] gh/kwen2501/246/base -> origin/gh/kwen2501/246/base 2025-10-10T01:23:14.6183173Z * [new branch] gh/kwen2501/246/head -> origin/gh/kwen2501/246/head 2025-10-10T01:23:14.6184904Z * [new branch] gh/kwen2501/246/orig -> origin/gh/kwen2501/246/orig 2025-10-10T01:23:14.6187239Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-10-10T01:23:14.6188984Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-10-10T01:23:14.6191184Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-10-10T01:23:14.6193619Z * [new branch] gh/kwen2501/248/base -> origin/gh/kwen2501/248/base 2025-10-10T01:23:14.6195259Z * [new branch] gh/kwen2501/248/head -> origin/gh/kwen2501/248/head 2025-10-10T01:23:14.6197279Z * [new branch] gh/kwen2501/248/orig -> origin/gh/kwen2501/248/orig 2025-10-10T01:23:14.6199626Z * [new branch] gh/kwen2501/249/base -> origin/gh/kwen2501/249/base 2025-10-10T01:23:14.6201320Z * [new branch] gh/kwen2501/249/head -> origin/gh/kwen2501/249/head 2025-10-10T01:23:14.6202966Z * [new branch] gh/kwen2501/249/orig -> origin/gh/kwen2501/249/orig 2025-10-10T01:23:14.6206174Z * [new branch] gh/kwen2501/250/base -> origin/gh/kwen2501/250/base 2025-10-10T01:23:14.6207386Z * [new branch] gh/kwen2501/250/head -> origin/gh/kwen2501/250/head 2025-10-10T01:23:14.6209187Z * [new branch] gh/kwen2501/250/orig -> origin/gh/kwen2501/250/orig 2025-10-10T01:23:14.6211526Z * [new branch] gh/kwen2501/251/base -> origin/gh/kwen2501/251/base 2025-10-10T01:23:14.6213432Z * [new branch] gh/kwen2501/251/head -> origin/gh/kwen2501/251/head 2025-10-10T01:23:14.6215091Z * [new branch] gh/kwen2501/251/orig -> origin/gh/kwen2501/251/orig 2025-10-10T01:23:14.6217453Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-10-10T01:23:14.6219108Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-10-10T01:23:14.6220781Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-10-10T01:23:14.6223142Z * [new branch] gh/kwen2501/253/base -> origin/gh/kwen2501/253/base 2025-10-10T01:23:14.6224893Z * [new branch] gh/kwen2501/253/head -> origin/gh/kwen2501/253/head 2025-10-10T01:23:14.6226596Z * [new branch] gh/kwen2501/253/orig -> origin/gh/kwen2501/253/orig 2025-10-10T01:23:14.6229037Z * [new branch] gh/kwen2501/254/base -> origin/gh/kwen2501/254/base 2025-10-10T01:23:14.6230722Z * [new branch] gh/kwen2501/254/head -> origin/gh/kwen2501/254/head 2025-10-10T01:23:14.6232439Z * [new branch] gh/kwen2501/254/orig -> origin/gh/kwen2501/254/orig 2025-10-10T01:23:14.6234844Z * [new branch] gh/kwen2501/255/base -> origin/gh/kwen2501/255/base 2025-10-10T01:23:14.6236504Z * [new branch] gh/kwen2501/255/head -> origin/gh/kwen2501/255/head 2025-10-10T01:23:14.6238338Z * [new branch] gh/kwen2501/255/orig -> origin/gh/kwen2501/255/orig 2025-10-10T01:23:14.6240937Z * [new branch] gh/kwen2501/256/base -> origin/gh/kwen2501/256/base 2025-10-10T01:23:14.6242712Z * [new branch] gh/kwen2501/256/head -> origin/gh/kwen2501/256/head 2025-10-10T01:23:14.6244338Z * [new branch] gh/kwen2501/256/orig -> origin/gh/kwen2501/256/orig 2025-10-10T01:23:14.6246776Z * [new branch] gh/kwen2501/257/base -> origin/gh/kwen2501/257/base 2025-10-10T01:23:14.6248518Z * [new branch] gh/kwen2501/257/head -> origin/gh/kwen2501/257/head 2025-10-10T01:23:14.6250185Z * [new branch] gh/kwen2501/257/orig -> origin/gh/kwen2501/257/orig 2025-10-10T01:23:14.6252593Z * [new branch] gh/kwen2501/258/base -> origin/gh/kwen2501/258/base 2025-10-10T01:23:14.6254277Z * [new branch] gh/kwen2501/258/head -> origin/gh/kwen2501/258/head 2025-10-10T01:23:14.6255978Z * [new branch] gh/kwen2501/258/orig -> origin/gh/kwen2501/258/orig 2025-10-10T01:23:14.6258310Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-10-10T01:23:14.6259974Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-10-10T01:23:14.6261743Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-10-10T01:23:14.6264112Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-10-10T01:23:14.6265801Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-10-10T01:23:14.6267456Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-10-10T01:23:14.6269799Z * [new branch] gh/kwen2501/261/base -> origin/gh/kwen2501/261/base 2025-10-10T01:23:14.6271515Z * [new branch] gh/kwen2501/261/head -> origin/gh/kwen2501/261/head 2025-10-10T01:23:14.6273225Z * [new branch] gh/kwen2501/261/orig -> origin/gh/kwen2501/261/orig 2025-10-10T01:23:14.6275633Z * [new branch] gh/kwen2501/262/base -> origin/gh/kwen2501/262/base 2025-10-10T01:23:14.6277348Z * [new branch] gh/kwen2501/262/head -> origin/gh/kwen2501/262/head 2025-10-10T01:23:14.6279332Z * [new branch] gh/kwen2501/262/orig -> origin/gh/kwen2501/262/orig 2025-10-10T01:23:14.6281450Z * [new branch] gh/kwen2501/263/base -> origin/gh/kwen2501/263/base 2025-10-10T01:23:14.6283172Z * [new branch] gh/kwen2501/263/head -> origin/gh/kwen2501/263/head 2025-10-10T01:23:14.6284839Z * [new branch] gh/kwen2501/263/orig -> origin/gh/kwen2501/263/orig 2025-10-10T01:23:14.6287173Z * [new branch] gh/kwen2501/264/base -> origin/gh/kwen2501/264/base 2025-10-10T01:23:14.6289307Z * [new branch] gh/kwen2501/264/head -> origin/gh/kwen2501/264/head 2025-10-10T01:23:14.6290909Z * [new branch] gh/kwen2501/264/orig -> origin/gh/kwen2501/264/orig 2025-10-10T01:23:14.6293348Z * [new branch] gh/kwen2501/265/base -> origin/gh/kwen2501/265/base 2025-10-10T01:23:14.6295056Z * [new branch] gh/kwen2501/265/head -> origin/gh/kwen2501/265/head 2025-10-10T01:23:14.6296722Z * [new branch] gh/kwen2501/265/orig -> origin/gh/kwen2501/265/orig 2025-10-10T01:23:14.6301031Z * [new branch] gh/kwen2501/266/base -> origin/gh/kwen2501/266/base 2025-10-10T01:23:14.6302788Z * [new branch] gh/kwen2501/266/head -> origin/gh/kwen2501/266/head 2025-10-10T01:23:14.6304558Z * [new branch] gh/kwen2501/266/orig -> origin/gh/kwen2501/266/orig 2025-10-10T01:23:14.6306924Z * [new branch] gh/kwen2501/267/base -> origin/gh/kwen2501/267/base 2025-10-10T01:23:14.6308639Z * [new branch] gh/kwen2501/267/head -> origin/gh/kwen2501/267/head 2025-10-10T01:23:14.6310331Z * [new branch] gh/kwen2501/267/orig -> origin/gh/kwen2501/267/orig 2025-10-10T01:23:14.6312539Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-10-10T01:23:14.6314210Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-10-10T01:23:14.6315975Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-10-10T01:23:14.6318448Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-10-10T01:23:14.6320411Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-10-10T01:23:14.6322097Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-10-10T01:23:14.6324535Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-10-10T01:23:14.6326329Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-10-10T01:23:14.6327991Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-10-10T01:23:14.6330405Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-10-10T01:23:14.6332101Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-10-10T01:23:14.6333815Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-10-10T01:23:14.6336146Z * [new branch] gh/kwen2501/272/base -> origin/gh/kwen2501/272/base 2025-10-10T01:23:14.6337836Z * [new branch] gh/kwen2501/272/head -> origin/gh/kwen2501/272/head 2025-10-10T01:23:14.6339518Z * [new branch] gh/kwen2501/272/orig -> origin/gh/kwen2501/272/orig 2025-10-10T01:23:14.6341893Z * [new branch] gh/kwen2501/273/base -> origin/gh/kwen2501/273/base 2025-10-10T01:23:14.6343578Z * [new branch] gh/kwen2501/273/head -> origin/gh/kwen2501/273/head 2025-10-10T01:23:14.6345242Z * [new branch] gh/kwen2501/273/orig -> origin/gh/kwen2501/273/orig 2025-10-10T01:23:14.6347665Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-10-10T01:23:14.6349502Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-10-10T01:23:14.6351085Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-10-10T01:23:14.6353833Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-10-10T01:23:14.6355514Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-10-10T01:23:14.6357223Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-10-10T01:23:14.6359850Z * [new branch] gh/laithsakka/262/base -> origin/gh/laithsakka/262/base 2025-10-10T01:23:14.6361453Z * [new branch] gh/laithsakka/262/head -> origin/gh/laithsakka/262/head 2025-10-10T01:23:14.6363245Z * [new branch] gh/laithsakka/262/orig -> origin/gh/laithsakka/262/orig 2025-10-10T01:23:14.6365603Z * [new branch] gh/laithsakka/263/base -> origin/gh/laithsakka/263/base 2025-10-10T01:23:14.6367258Z * [new branch] gh/laithsakka/263/head -> origin/gh/laithsakka/263/head 2025-10-10T01:23:14.6368933Z * [new branch] gh/laithsakka/263/orig -> origin/gh/laithsakka/263/orig 2025-10-10T01:23:14.6371180Z * [new branch] gh/laithsakka/264/base -> origin/gh/laithsakka/264/base 2025-10-10T01:23:14.6372983Z * [new branch] gh/laithsakka/264/head -> origin/gh/laithsakka/264/head 2025-10-10T01:23:14.6374644Z * [new branch] gh/laithsakka/264/orig -> origin/gh/laithsakka/264/orig 2025-10-10T01:23:14.6376939Z * [new branch] gh/laithsakka/268/base -> origin/gh/laithsakka/268/base 2025-10-10T01:23:14.6378612Z * [new branch] gh/laithsakka/268/head -> origin/gh/laithsakka/268/head 2025-10-10T01:23:14.6380279Z * [new branch] gh/laithsakka/268/orig -> origin/gh/laithsakka/268/orig 2025-10-10T01:23:14.6382526Z * [new branch] gh/laithsakka/269/base -> origin/gh/laithsakka/269/base 2025-10-10T01:23:14.6384208Z * [new branch] gh/laithsakka/269/head -> origin/gh/laithsakka/269/head 2025-10-10T01:23:14.6385871Z * [new branch] gh/laithsakka/269/orig -> origin/gh/laithsakka/269/orig 2025-10-10T01:23:14.6388189Z * [new branch] gh/laithsakka/271/base -> origin/gh/laithsakka/271/base 2025-10-10T01:23:14.6389847Z * [new branch] gh/laithsakka/271/head -> origin/gh/laithsakka/271/head 2025-10-10T01:23:14.6391523Z * [new branch] gh/laithsakka/271/orig -> origin/gh/laithsakka/271/orig 2025-10-10T01:23:14.6393883Z * [new branch] gh/laithsakka/272/base -> origin/gh/laithsakka/272/base 2025-10-10T01:23:14.6395583Z * [new branch] gh/laithsakka/272/head -> origin/gh/laithsakka/272/head 2025-10-10T01:23:14.6397521Z * [new branch] gh/laithsakka/272/orig -> origin/gh/laithsakka/272/orig 2025-10-10T01:23:14.6399895Z * [new branch] gh/laithsakka/273/base -> origin/gh/laithsakka/273/base 2025-10-10T01:23:14.6401558Z * [new branch] gh/laithsakka/273/head -> origin/gh/laithsakka/273/head 2025-10-10T01:23:14.6403285Z * [new branch] gh/laithsakka/273/orig -> origin/gh/laithsakka/273/orig 2025-10-10T01:23:14.6405520Z * [new branch] gh/laithsakka/274/base -> origin/gh/laithsakka/274/base 2025-10-10T01:23:14.6407208Z * [new branch] gh/laithsakka/274/head -> origin/gh/laithsakka/274/head 2025-10-10T01:23:14.6408942Z * [new branch] gh/laithsakka/274/orig -> origin/gh/laithsakka/274/orig 2025-10-10T01:23:14.6411153Z * [new branch] gh/laithsakka/275/base -> origin/gh/laithsakka/275/base 2025-10-10T01:23:14.6412761Z * [new branch] gh/laithsakka/275/head -> origin/gh/laithsakka/275/head 2025-10-10T01:23:14.6414467Z * [new branch] gh/laithsakka/275/orig -> origin/gh/laithsakka/275/orig 2025-10-10T01:23:14.6417038Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-10-10T01:23:14.6418585Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-10-10T01:23:14.6420239Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-10-10T01:23:14.6422688Z * [new branch] gh/laithsakka/277/base -> origin/gh/laithsakka/277/base 2025-10-10T01:23:14.6424458Z * [new branch] gh/laithsakka/277/head -> origin/gh/laithsakka/277/head 2025-10-10T01:23:14.6426145Z * [new branch] gh/laithsakka/277/orig -> origin/gh/laithsakka/277/orig 2025-10-10T01:23:14.6428312Z * [new branch] gh/laithsakka/278/base -> origin/gh/laithsakka/278/base 2025-10-10T01:23:14.6430053Z * [new branch] gh/laithsakka/278/head -> origin/gh/laithsakka/278/head 2025-10-10T01:23:14.6431827Z * [new branch] gh/laithsakka/278/orig -> origin/gh/laithsakka/278/orig 2025-10-10T01:23:14.6434629Z * [new branch] gh/laithsakka/279/base -> origin/gh/laithsakka/279/base 2025-10-10T01:23:14.6436257Z * [new branch] gh/laithsakka/279/head -> origin/gh/laithsakka/279/head 2025-10-10T01:23:14.6438000Z * [new branch] gh/laithsakka/279/orig -> origin/gh/laithsakka/279/orig 2025-10-10T01:23:14.6441107Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-10-10T01:23:14.6443545Z * [new branch] gh/laithsakka/280/base -> origin/gh/laithsakka/280/base 2025-10-10T01:23:14.6446195Z * [new branch] gh/laithsakka/280/head -> origin/gh/laithsakka/280/head 2025-10-10T01:23:14.6447951Z * [new branch] gh/laithsakka/280/orig -> origin/gh/laithsakka/280/orig 2025-10-10T01:23:14.6450604Z * [new branch] gh/laithsakka/281/base -> origin/gh/laithsakka/281/base 2025-10-10T01:23:14.6452324Z * [new branch] gh/laithsakka/281/head -> origin/gh/laithsakka/281/head 2025-10-10T01:23:14.6454030Z * [new branch] gh/laithsakka/281/orig -> origin/gh/laithsakka/281/orig 2025-10-10T01:23:14.6456299Z * [new branch] gh/laithsakka/282/base -> origin/gh/laithsakka/282/base 2025-10-10T01:23:14.6458090Z * [new branch] gh/laithsakka/282/head -> origin/gh/laithsakka/282/head 2025-10-10T01:23:14.6459862Z * [new branch] gh/laithsakka/282/orig -> origin/gh/laithsakka/282/orig 2025-10-10T01:23:14.6462291Z * [new branch] gh/laithsakka/283/base -> origin/gh/laithsakka/283/base 2025-10-10T01:23:14.6464068Z * [new branch] gh/laithsakka/283/head -> origin/gh/laithsakka/283/head 2025-10-10T01:23:14.6465787Z * [new branch] gh/laithsakka/283/orig -> origin/gh/laithsakka/283/orig 2025-10-10T01:23:14.6468189Z * [new branch] gh/laithsakka/284/base -> origin/gh/laithsakka/284/base 2025-10-10T01:23:14.6469840Z * [new branch] gh/laithsakka/284/head -> origin/gh/laithsakka/284/head 2025-10-10T01:23:14.6471561Z * [new branch] gh/laithsakka/284/orig -> origin/gh/laithsakka/284/orig 2025-10-10T01:23:14.6473955Z * [new branch] gh/laithsakka/285/base -> origin/gh/laithsakka/285/base 2025-10-10T01:23:14.6475688Z * [new branch] gh/laithsakka/285/head -> origin/gh/laithsakka/285/head 2025-10-10T01:23:14.6477375Z * [new branch] gh/laithsakka/285/orig -> origin/gh/laithsakka/285/orig 2025-10-10T01:23:14.6480518Z * [new branch] gh/laithsakka/286/base -> origin/gh/laithsakka/286/base 2025-10-10T01:23:14.6482194Z * [new branch] gh/laithsakka/286/head -> origin/gh/laithsakka/286/head 2025-10-10T01:23:14.6483868Z * [new branch] gh/laithsakka/286/orig -> origin/gh/laithsakka/286/orig 2025-10-10T01:23:14.6486350Z * [new branch] gh/laithsakka/287/base -> origin/gh/laithsakka/287/base 2025-10-10T01:23:14.6488054Z * [new branch] gh/laithsakka/287/head -> origin/gh/laithsakka/287/head 2025-10-10T01:23:14.6489713Z * [new branch] gh/laithsakka/287/orig -> origin/gh/laithsakka/287/orig 2025-10-10T01:23:14.6492070Z * [new branch] gh/laithsakka/288/base -> origin/gh/laithsakka/288/base 2025-10-10T01:23:14.6493766Z * [new branch] gh/laithsakka/288/head -> origin/gh/laithsakka/288/head 2025-10-10T01:23:14.6495474Z * [new branch] gh/laithsakka/288/orig -> origin/gh/laithsakka/288/orig 2025-10-10T01:23:14.6498292Z * [new branch] gh/laithsakka/289/base -> origin/gh/laithsakka/289/base 2025-10-10T01:23:14.6500102Z * [new branch] gh/laithsakka/289/head -> origin/gh/laithsakka/289/head 2025-10-10T01:23:14.6501841Z * [new branch] gh/laithsakka/289/orig -> origin/gh/laithsakka/289/orig 2025-10-10T01:23:14.6504148Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-10-10T01:23:14.6506358Z * [new branch] gh/laithsakka/290/base -> origin/gh/laithsakka/290/base 2025-10-10T01:23:14.6508247Z * [new branch] gh/laithsakka/290/head -> origin/gh/laithsakka/290/head 2025-10-10T01:23:14.6509952Z * [new branch] gh/laithsakka/290/orig -> origin/gh/laithsakka/290/orig 2025-10-10T01:23:14.6512793Z * [new branch] gh/laithsakka/291/base -> origin/gh/laithsakka/291/base 2025-10-10T01:23:14.6514461Z * [new branch] gh/laithsakka/291/head -> origin/gh/laithsakka/291/head 2025-10-10T01:23:14.6516106Z * [new branch] gh/laithsakka/291/orig -> origin/gh/laithsakka/291/orig 2025-10-10T01:23:14.6518616Z * [new branch] gh/laithsakka/292/base -> origin/gh/laithsakka/292/base 2025-10-10T01:23:14.6521105Z * [new branch] gh/laithsakka/292/head -> origin/gh/laithsakka/292/head 2025-10-10T01:23:14.6522693Z * [new branch] gh/laithsakka/292/orig -> origin/gh/laithsakka/292/orig 2025-10-10T01:23:14.6524918Z * [new branch] gh/laithsakka/293/base -> origin/gh/laithsakka/293/base 2025-10-10T01:23:14.6526613Z * [new branch] gh/laithsakka/293/head -> origin/gh/laithsakka/293/head 2025-10-10T01:23:14.6528308Z * [new branch] gh/laithsakka/293/orig -> origin/gh/laithsakka/293/orig 2025-10-10T01:23:14.6530738Z * [new branch] gh/laithsakka/294/base -> origin/gh/laithsakka/294/base 2025-10-10T01:23:14.6532533Z * [new branch] gh/laithsakka/294/head -> origin/gh/laithsakka/294/head 2025-10-10T01:23:14.6534275Z * [new branch] gh/laithsakka/294/orig -> origin/gh/laithsakka/294/orig 2025-10-10T01:23:14.6536801Z * [new branch] gh/laithsakka/295/base -> origin/gh/laithsakka/295/base 2025-10-10T01:23:14.6538405Z * [new branch] gh/laithsakka/295/head -> origin/gh/laithsakka/295/head 2025-10-10T01:23:14.6540055Z * [new branch] gh/laithsakka/295/orig -> origin/gh/laithsakka/295/orig 2025-10-10T01:23:14.6542432Z * [new branch] gh/laithsakka/296/base -> origin/gh/laithsakka/296/base 2025-10-10T01:23:14.6544248Z * [new branch] gh/laithsakka/296/head -> origin/gh/laithsakka/296/head 2025-10-10T01:23:14.6545853Z * [new branch] gh/laithsakka/296/orig -> origin/gh/laithsakka/296/orig 2025-10-10T01:23:14.6548290Z * [new branch] gh/laithsakka/297/base -> origin/gh/laithsakka/297/base 2025-10-10T01:23:14.6549919Z * [new branch] gh/laithsakka/297/head -> origin/gh/laithsakka/297/head 2025-10-10T01:23:14.6551639Z * [new branch] gh/laithsakka/297/orig -> origin/gh/laithsakka/297/orig 2025-10-10T01:23:14.6554013Z * [new branch] gh/laithsakka/298/base -> origin/gh/laithsakka/298/base 2025-10-10T01:23:14.6555874Z * [new branch] gh/laithsakka/298/head -> origin/gh/laithsakka/298/head 2025-10-10T01:23:14.6557440Z * [new branch] gh/laithsakka/298/orig -> origin/gh/laithsakka/298/orig 2025-10-10T01:23:14.6559949Z * [new branch] gh/laithsakka/299/base -> origin/gh/laithsakka/299/base 2025-10-10T01:23:14.6561627Z * [new branch] gh/laithsakka/299/head -> origin/gh/laithsakka/299/head 2025-10-10T01:23:14.6563350Z * [new branch] gh/laithsakka/299/orig -> origin/gh/laithsakka/299/orig 2025-10-10T01:23:14.6565572Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-10-10T01:23:14.6567275Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-10-10T01:23:14.6570068Z * [new branch] gh/laithsakka/300/base -> origin/gh/laithsakka/300/base 2025-10-10T01:23:14.6572259Z * [new branch] gh/laithsakka/300/head -> origin/gh/laithsakka/300/head 2025-10-10T01:23:14.6573942Z * [new branch] gh/laithsakka/300/orig -> origin/gh/laithsakka/300/orig 2025-10-10T01:23:14.6576411Z * [new branch] gh/laithsakka/301/base -> origin/gh/laithsakka/301/base 2025-10-10T01:23:14.6578133Z * [new branch] gh/laithsakka/301/head -> origin/gh/laithsakka/301/head 2025-10-10T01:23:14.6579840Z * [new branch] gh/laithsakka/301/orig -> origin/gh/laithsakka/301/orig 2025-10-10T01:23:14.6582275Z * [new branch] gh/laithsakka/302/base -> origin/gh/laithsakka/302/base 2025-10-10T01:23:14.6583960Z * [new branch] gh/laithsakka/302/head -> origin/gh/laithsakka/302/head 2025-10-10T01:23:14.6585544Z * [new branch] gh/laithsakka/302/orig -> origin/gh/laithsakka/302/orig 2025-10-10T01:23:14.6587787Z * [new branch] gh/laithsakka/303/base -> origin/gh/laithsakka/303/base 2025-10-10T01:23:14.6589545Z * [new branch] gh/laithsakka/303/head -> origin/gh/laithsakka/303/head 2025-10-10T01:23:14.6591148Z * [new branch] gh/laithsakka/303/orig -> origin/gh/laithsakka/303/orig 2025-10-10T01:23:14.6593497Z * [new branch] gh/laithsakka/304/base -> origin/gh/laithsakka/304/base 2025-10-10T01:23:14.6595150Z * [new branch] gh/laithsakka/304/head -> origin/gh/laithsakka/304/head 2025-10-10T01:23:14.6596948Z * [new branch] gh/laithsakka/304/orig -> origin/gh/laithsakka/304/orig 2025-10-10T01:23:14.6599209Z * [new branch] gh/laithsakka/305/base -> origin/gh/laithsakka/305/base 2025-10-10T01:23:14.6600953Z * [new branch] gh/laithsakka/305/head -> origin/gh/laithsakka/305/head 2025-10-10T01:23:14.6602622Z * [new branch] gh/laithsakka/305/orig -> origin/gh/laithsakka/305/orig 2025-10-10T01:23:14.6605026Z * [new branch] gh/laithsakka/306/base -> origin/gh/laithsakka/306/base 2025-10-10T01:23:14.6606669Z * [new branch] gh/laithsakka/306/head -> origin/gh/laithsakka/306/head 2025-10-10T01:23:14.6608276Z * [new branch] gh/laithsakka/306/orig -> origin/gh/laithsakka/306/orig 2025-10-10T01:23:14.6610503Z * [new branch] gh/laithsakka/307/base -> origin/gh/laithsakka/307/base 2025-10-10T01:23:14.6612284Z * [new branch] gh/laithsakka/307/head -> origin/gh/laithsakka/307/head 2025-10-10T01:23:14.6613845Z * [new branch] gh/laithsakka/307/orig -> origin/gh/laithsakka/307/orig 2025-10-10T01:23:14.6616108Z * [new branch] gh/laithsakka/308/base -> origin/gh/laithsakka/308/base 2025-10-10T01:23:14.6617789Z * [new branch] gh/laithsakka/308/head -> origin/gh/laithsakka/308/head 2025-10-10T01:23:14.6619458Z * [new branch] gh/laithsakka/308/orig -> origin/gh/laithsakka/308/orig 2025-10-10T01:23:14.6621858Z * [new branch] gh/laithsakka/309/base -> origin/gh/laithsakka/309/base 2025-10-10T01:23:14.6623433Z * [new branch] gh/laithsakka/309/head -> origin/gh/laithsakka/309/head 2025-10-10T01:23:14.6625213Z * [new branch] gh/laithsakka/309/orig -> origin/gh/laithsakka/309/orig 2025-10-10T01:23:14.6627509Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-10-10T01:23:14.6629133Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-10-10T01:23:14.6631517Z * [new branch] gh/laithsakka/310/base -> origin/gh/laithsakka/310/base 2025-10-10T01:23:14.6633256Z * [new branch] gh/laithsakka/310/head -> origin/gh/laithsakka/310/head 2025-10-10T01:23:14.6634859Z * [new branch] gh/laithsakka/310/orig -> origin/gh/laithsakka/310/orig 2025-10-10T01:23:14.6637259Z * [new branch] gh/laithsakka/311/base -> origin/gh/laithsakka/311/base 2025-10-10T01:23:14.6638976Z * [new branch] gh/laithsakka/311/head -> origin/gh/laithsakka/311/head 2025-10-10T01:23:14.6640901Z * [new branch] gh/laithsakka/311/orig -> origin/gh/laithsakka/311/orig 2025-10-10T01:23:14.6643216Z * [new branch] gh/laithsakka/312/base -> origin/gh/laithsakka/312/base 2025-10-10T01:23:14.6645106Z * [new branch] gh/laithsakka/312/head -> origin/gh/laithsakka/312/head 2025-10-10T01:23:14.6647229Z * [new branch] gh/laithsakka/312/orig -> origin/gh/laithsakka/312/orig 2025-10-10T01:23:14.6649490Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-10-10T01:23:14.6651207Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-10-10T01:23:14.6652981Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-10-10T01:23:14.6655234Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-10-10T01:23:14.6656927Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-10-10T01:23:14.6659834Z * [new branch] gh/liangel/1/base -> origin/gh/liangel/1/base 2025-10-10T01:23:14.6661422Z * [new branch] gh/liangel/1/head -> origin/gh/liangel/1/head 2025-10-10T01:23:14.6663743Z * [new branch] gh/liangel/1/orig -> origin/gh/liangel/1/orig 2025-10-10T01:23:14.6666152Z * [new branch] gh/liangel/2/base -> origin/gh/liangel/2/base 2025-10-10T01:23:14.6667776Z * [new branch] gh/liangel/2/head -> origin/gh/liangel/2/head 2025-10-10T01:23:14.6669480Z * [new branch] gh/liangel/2/orig -> origin/gh/liangel/2/orig 2025-10-10T01:23:14.6671661Z * [new branch] gh/liangel/3/base -> origin/gh/liangel/3/base 2025-10-10T01:23:14.6673549Z * [new branch] gh/liangel/3/head -> origin/gh/liangel/3/head 2025-10-10T01:23:14.6675194Z * [new branch] gh/liangel/3/orig -> origin/gh/liangel/3/orig 2025-10-10T01:23:14.6677538Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-10-10T01:23:14.6679293Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-10-10T01:23:14.6681036Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-10-10T01:23:14.6685236Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-10-10T01:23:14.6686923Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-10-10T01:23:14.6689615Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-10-10T01:23:14.6691352Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-10-10T01:23:14.6693161Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-10-10T01:23:14.6695437Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-10-10T01:23:14.6698763Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-10-10T01:23:14.6700491Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-10-10T01:23:14.6702831Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-10-10T01:23:14.6704491Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-10-10T01:23:14.6706188Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-10-10T01:23:14.6708475Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-10-10T01:23:14.6710118Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-10-10T01:23:14.6711754Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-10-10T01:23:14.6714487Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-10-10T01:23:14.6716763Z * [new branch] gh/malfet/396/base -> origin/gh/malfet/396/base 2025-10-10T01:23:14.6718531Z * [new branch] gh/malfet/396/head -> origin/gh/malfet/396/head 2025-10-10T01:23:14.6720333Z * [new branch] gh/malfet/396/orig -> origin/gh/malfet/396/orig 2025-10-10T01:23:14.6722672Z * [new branch] gh/malfet/397/base -> origin/gh/malfet/397/base 2025-10-10T01:23:14.6724415Z * [new branch] gh/malfet/397/head -> origin/gh/malfet/397/head 2025-10-10T01:23:14.6726092Z * [new branch] gh/malfet/397/orig -> origin/gh/malfet/397/orig 2025-10-10T01:23:14.6728413Z * [new branch] gh/malfet/398/base -> origin/gh/malfet/398/base 2025-10-10T01:23:14.6730037Z * [new branch] gh/malfet/398/head -> origin/gh/malfet/398/head 2025-10-10T01:23:14.6731791Z * [new branch] gh/malfet/398/orig -> origin/gh/malfet/398/orig 2025-10-10T01:23:14.6734056Z * [new branch] gh/malfet/399/base -> origin/gh/malfet/399/base 2025-10-10T01:23:14.6735753Z * [new branch] gh/malfet/399/head -> origin/gh/malfet/399/head 2025-10-10T01:23:14.6737419Z * [new branch] gh/malfet/399/orig -> origin/gh/malfet/399/orig 2025-10-10T01:23:14.6739846Z * [new branch] gh/malfet/414/base -> origin/gh/malfet/414/base 2025-10-10T01:23:14.6741569Z * [new branch] gh/malfet/414/head -> origin/gh/malfet/414/head 2025-10-10T01:23:14.6743250Z * [new branch] gh/malfet/414/orig -> origin/gh/malfet/414/orig 2025-10-10T01:23:14.6745598Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-10-10T01:23:14.6747335Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-10-10T01:23:14.6749007Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-10-10T01:23:14.6751287Z * [new branch] gh/malfet/418/base -> origin/gh/malfet/418/base 2025-10-10T01:23:14.6752910Z * [new branch] gh/malfet/418/head -> origin/gh/malfet/418/head 2025-10-10T01:23:14.6754607Z * [new branch] gh/malfet/418/orig -> origin/gh/malfet/418/orig 2025-10-10T01:23:14.6756958Z * [new branch] gh/malfet/505/base -> origin/gh/malfet/505/base 2025-10-10T01:23:14.6758642Z * [new branch] gh/malfet/505/head -> origin/gh/malfet/505/head 2025-10-10T01:23:14.6760538Z * [new branch] gh/malfet/505/orig -> origin/gh/malfet/505/orig 2025-10-10T01:23:14.6762691Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-10-10T01:23:14.6764572Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-10-10T01:23:14.6766176Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-10-10T01:23:14.6768489Z * [new branch] gh/malfet/507/base -> origin/gh/malfet/507/base 2025-10-10T01:23:14.6770162Z * [new branch] gh/malfet/507/head -> origin/gh/malfet/507/head 2025-10-10T01:23:14.6771901Z * [new branch] gh/malfet/507/orig -> origin/gh/malfet/507/orig 2025-10-10T01:23:14.6774197Z * [new branch] gh/malfet/513/base -> origin/gh/malfet/513/base 2025-10-10T01:23:14.6775879Z * [new branch] gh/malfet/513/head -> origin/gh/malfet/513/head 2025-10-10T01:23:14.6777555Z * [new branch] gh/malfet/513/orig -> origin/gh/malfet/513/orig 2025-10-10T01:23:14.6779822Z * [new branch] gh/malfet/516/base -> origin/gh/malfet/516/base 2025-10-10T01:23:14.6781521Z * [new branch] gh/malfet/516/head -> origin/gh/malfet/516/head 2025-10-10T01:23:14.6783374Z * [new branch] gh/malfet/516/orig -> origin/gh/malfet/516/orig 2025-10-10T01:23:14.6785508Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-10-10T01:23:14.6787279Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-10-10T01:23:14.6789533Z * [new branch] gh/malfet/518/base -> origin/gh/malfet/518/base 2025-10-10T01:23:14.6791250Z * [new branch] gh/malfet/518/head -> origin/gh/malfet/518/head 2025-10-10T01:23:14.6793069Z * [new branch] gh/malfet/518/orig -> origin/gh/malfet/518/orig 2025-10-10T01:23:14.6795311Z * [new branch] gh/malfet/519/base -> origin/gh/malfet/519/base 2025-10-10T01:23:14.6797221Z * [new branch] gh/malfet/519/head -> origin/gh/malfet/519/head 2025-10-10T01:23:14.6799038Z * [new branch] gh/malfet/519/orig -> origin/gh/malfet/519/orig 2025-10-10T01:23:14.6801382Z * [new branch] gh/malfet/520/base -> origin/gh/malfet/520/base 2025-10-10T01:23:14.6803145Z * [new branch] gh/malfet/520/head -> origin/gh/malfet/520/head 2025-10-10T01:23:14.6804798Z * [new branch] gh/malfet/520/orig -> origin/gh/malfet/520/orig 2025-10-10T01:23:14.6807474Z * [new branch] gh/malfet/521/base -> origin/gh/malfet/521/base 2025-10-10T01:23:14.6809207Z * [new branch] gh/malfet/521/head -> origin/gh/malfet/521/head 2025-10-10T01:23:14.6811016Z * [new branch] gh/malfet/521/orig -> origin/gh/malfet/521/orig 2025-10-10T01:23:14.6813380Z * [new branch] gh/malfet/522/base -> origin/gh/malfet/522/base 2025-10-10T01:23:14.6815101Z * [new branch] gh/malfet/522/head -> origin/gh/malfet/522/head 2025-10-10T01:23:14.6816792Z * [new branch] gh/malfet/522/orig -> origin/gh/malfet/522/orig 2025-10-10T01:23:14.6819255Z * [new branch] gh/malfet/523/base -> origin/gh/malfet/523/base 2025-10-10T01:23:14.6820952Z * [new branch] gh/malfet/523/head -> origin/gh/malfet/523/head 2025-10-10T01:23:14.6822768Z * [new branch] gh/malfet/523/orig -> origin/gh/malfet/523/orig 2025-10-10T01:23:14.6825091Z * [new branch] gh/malfet/524/base -> origin/gh/malfet/524/base 2025-10-10T01:23:14.6826760Z * [new branch] gh/malfet/524/head -> origin/gh/malfet/524/head 2025-10-10T01:23:14.6828460Z * [new branch] gh/malfet/524/orig -> origin/gh/malfet/524/orig 2025-10-10T01:23:14.6830894Z * [new branch] gh/malfet/525/base -> origin/gh/malfet/525/base 2025-10-10T01:23:14.6832655Z * [new branch] gh/malfet/525/head -> origin/gh/malfet/525/head 2025-10-10T01:23:14.6834576Z * [new branch] gh/malfet/525/orig -> origin/gh/malfet/525/orig 2025-10-10T01:23:14.6836830Z * [new branch] gh/malfet/526/base -> origin/gh/malfet/526/base 2025-10-10T01:23:14.6838499Z * [new branch] gh/malfet/526/head -> origin/gh/malfet/526/head 2025-10-10T01:23:14.6840408Z * [new branch] gh/malfet/526/orig -> origin/gh/malfet/526/orig 2025-10-10T01:23:14.6842670Z * [new branch] gh/malfet/527/base -> origin/gh/malfet/527/base 2025-10-10T01:23:14.6844355Z * [new branch] gh/malfet/527/head -> origin/gh/malfet/527/head 2025-10-10T01:23:14.6846052Z * [new branch] gh/malfet/527/orig -> origin/gh/malfet/527/orig 2025-10-10T01:23:14.6849000Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-10-10T01:23:14.6850717Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-10-10T01:23:14.6852371Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-10-10T01:23:14.6854749Z * [new branch] gh/malfet/529/base -> origin/gh/malfet/529/base 2025-10-10T01:23:14.6856451Z * [new branch] gh/malfet/529/head -> origin/gh/malfet/529/head 2025-10-10T01:23:14.6858257Z * [new branch] gh/malfet/529/orig -> origin/gh/malfet/529/orig 2025-10-10T01:23:14.6860582Z * [new branch] gh/malfet/530/base -> origin/gh/malfet/530/base 2025-10-10T01:23:14.6862312Z * [new branch] gh/malfet/530/head -> origin/gh/malfet/530/head 2025-10-10T01:23:14.6863990Z * [new branch] gh/malfet/530/orig -> origin/gh/malfet/530/orig 2025-10-10T01:23:14.6866374Z * [new branch] gh/malfet/531/base -> origin/gh/malfet/531/base 2025-10-10T01:23:14.6868041Z * [new branch] gh/malfet/531/head -> origin/gh/malfet/531/head 2025-10-10T01:23:14.6869829Z * [new branch] gh/malfet/531/orig -> origin/gh/malfet/531/orig 2025-10-10T01:23:14.6872172Z * [new branch] gh/malfet/532/base -> origin/gh/malfet/532/base 2025-10-10T01:23:14.6873828Z * [new branch] gh/malfet/532/head -> origin/gh/malfet/532/head 2025-10-10T01:23:14.6875593Z * [new branch] gh/malfet/532/orig -> origin/gh/malfet/532/orig 2025-10-10T01:23:14.6878415Z * [new branch] gh/malfet/533/base -> origin/gh/malfet/533/base 2025-10-10T01:23:14.6880242Z * [new branch] gh/malfet/533/head -> origin/gh/malfet/533/head 2025-10-10T01:23:14.6882006Z * [new branch] gh/malfet/533/orig -> origin/gh/malfet/533/orig 2025-10-10T01:23:14.6884414Z * [new branch] gh/malfet/534/base -> origin/gh/malfet/534/base 2025-10-10T01:23:14.6886521Z * [new branch] gh/malfet/534/head -> origin/gh/malfet/534/head 2025-10-10T01:23:14.6888271Z * [new branch] gh/malfet/534/orig -> origin/gh/malfet/534/orig 2025-10-10T01:23:14.6890658Z * [new branch] gh/malfet/535/base -> origin/gh/malfet/535/base 2025-10-10T01:23:14.6892337Z * [new branch] gh/malfet/535/head -> origin/gh/malfet/535/head 2025-10-10T01:23:14.6894041Z * [new branch] gh/malfet/535/orig -> origin/gh/malfet/535/orig 2025-10-10T01:23:14.6896579Z * [new branch] gh/malfet/536/base -> origin/gh/malfet/536/base 2025-10-10T01:23:14.6898307Z * [new branch] gh/malfet/536/head -> origin/gh/malfet/536/head 2025-10-10T01:23:14.6900022Z * [new branch] gh/malfet/536/orig -> origin/gh/malfet/536/orig 2025-10-10T01:23:14.6902309Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-10-10T01:23:14.6903986Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-10-10T01:23:14.6906011Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-10-10T01:23:14.6908328Z * [new branch] gh/malfet/538/base -> origin/gh/malfet/538/base 2025-10-10T01:23:14.6910060Z * [new branch] gh/malfet/538/head -> origin/gh/malfet/538/head 2025-10-10T01:23:14.6911761Z * [new branch] gh/malfet/538/orig -> origin/gh/malfet/538/orig 2025-10-10T01:23:14.6914046Z * [new branch] gh/malfet/539/base -> origin/gh/malfet/539/base 2025-10-10T01:23:14.6915808Z * [new branch] gh/malfet/539/head -> origin/gh/malfet/539/head 2025-10-10T01:23:14.6917459Z * [new branch] gh/malfet/539/orig -> origin/gh/malfet/539/orig 2025-10-10T01:23:14.6919915Z * [new branch] gh/malfet/540/base -> origin/gh/malfet/540/base 2025-10-10T01:23:14.6921654Z * [new branch] gh/malfet/540/head -> origin/gh/malfet/540/head 2025-10-10T01:23:14.6923886Z * [new branch] gh/malfet/540/orig -> origin/gh/malfet/540/orig 2025-10-10T01:23:14.6926190Z * [new branch] gh/malfet/541/base -> origin/gh/malfet/541/base 2025-10-10T01:23:14.6927908Z * [new branch] gh/malfet/541/head -> origin/gh/malfet/541/head 2025-10-10T01:23:14.6929651Z * [new branch] gh/malfet/541/orig -> origin/gh/malfet/541/orig 2025-10-10T01:23:14.6931989Z * [new branch] gh/malfet/542/base -> origin/gh/malfet/542/base 2025-10-10T01:23:14.6933776Z * [new branch] gh/malfet/542/head -> origin/gh/malfet/542/head 2025-10-10T01:23:14.6935477Z * [new branch] gh/malfet/542/orig -> origin/gh/malfet/542/orig 2025-10-10T01:23:14.6937891Z * [new branch] gh/malfet/543/base -> origin/gh/malfet/543/base 2025-10-10T01:23:14.6939553Z * [new branch] gh/malfet/543/head -> origin/gh/malfet/543/head 2025-10-10T01:23:14.6941155Z * [new branch] gh/malfet/543/orig -> origin/gh/malfet/543/orig 2025-10-10T01:23:14.6943607Z * [new branch] gh/malfet/544/base -> origin/gh/malfet/544/base 2025-10-10T01:23:14.6945368Z * [new branch] gh/malfet/544/head -> origin/gh/malfet/544/head 2025-10-10T01:23:14.6947136Z * [new branch] gh/malfet/544/orig -> origin/gh/malfet/544/orig 2025-10-10T01:23:14.6949441Z * [new branch] gh/malfet/545/base -> origin/gh/malfet/545/base 2025-10-10T01:23:14.6951084Z * [new branch] gh/malfet/545/head -> origin/gh/malfet/545/head 2025-10-10T01:23:14.6952893Z * [new branch] gh/malfet/545/orig -> origin/gh/malfet/545/orig 2025-10-10T01:23:14.6955248Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-10-10T01:23:14.6956915Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-10-10T01:23:14.6958568Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-10-10T01:23:14.6961037Z * [new branch] gh/malfet/547/base -> origin/gh/malfet/547/base 2025-10-10T01:23:14.6962684Z * [new branch] gh/malfet/547/head -> origin/gh/malfet/547/head 2025-10-10T01:23:14.6964340Z * [new branch] gh/malfet/547/orig -> origin/gh/malfet/547/orig 2025-10-10T01:23:14.6966694Z * [new branch] gh/malfet/548/base -> origin/gh/malfet/548/base 2025-10-10T01:23:14.6968375Z * [new branch] gh/malfet/548/head -> origin/gh/malfet/548/head 2025-10-10T01:23:14.6970152Z * [new branch] gh/malfet/548/orig -> origin/gh/malfet/548/orig 2025-10-10T01:23:14.6972548Z * [new branch] gh/malfet/549/base -> origin/gh/malfet/549/base 2025-10-10T01:23:14.6974343Z * [new branch] gh/malfet/549/head -> origin/gh/malfet/549/head 2025-10-10T01:23:14.6976121Z * [new branch] gh/malfet/549/orig -> origin/gh/malfet/549/orig 2025-10-10T01:23:14.6978414Z * [new branch] gh/malfet/550/base -> origin/gh/malfet/550/base 2025-10-10T01:23:14.6980211Z * [new branch] gh/malfet/550/head -> origin/gh/malfet/550/head 2025-10-10T01:23:14.6981878Z * [new branch] gh/malfet/550/orig -> origin/gh/malfet/550/orig 2025-10-10T01:23:14.6984174Z * [new branch] gh/malfet/551/base -> origin/gh/malfet/551/base 2025-10-10T01:23:14.6985905Z * [new branch] gh/malfet/551/head -> origin/gh/malfet/551/head 2025-10-10T01:23:14.6987563Z * [new branch] gh/malfet/551/orig -> origin/gh/malfet/551/orig 2025-10-10T01:23:14.6989905Z * [new branch] gh/malfet/552/base -> origin/gh/malfet/552/base 2025-10-10T01:23:14.6991615Z * [new branch] gh/malfet/552/head -> origin/gh/malfet/552/head 2025-10-10T01:23:14.6993306Z * [new branch] gh/malfet/552/orig -> origin/gh/malfet/552/orig 2025-10-10T01:23:14.6995639Z * [new branch] gh/malfet/553/base -> origin/gh/malfet/553/base 2025-10-10T01:23:14.6997487Z * [new branch] gh/malfet/553/head -> origin/gh/malfet/553/head 2025-10-10T01:23:14.6999306Z * [new branch] gh/malfet/553/orig -> origin/gh/malfet/553/orig 2025-10-10T01:23:14.7001818Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-10-10T01:23:14.7003498Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-10-10T01:23:14.7006327Z * [new branch] gh/manuelcandales/10/base -> origin/gh/manuelcandales/10/base 2025-10-10T01:23:14.7008064Z * [new branch] gh/manuelcandales/10/head -> origin/gh/manuelcandales/10/head 2025-10-10T01:23:14.7009734Z * [new branch] gh/manuelcandales/10/orig -> origin/gh/manuelcandales/10/orig 2025-10-10T01:23:14.7011947Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-10-10T01:23:14.7013624Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-10-10T01:23:14.7015333Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-10-10T01:23:14.7017565Z * [new branch] gh/manuelcandales/9/base -> origin/gh/manuelcandales/9/base 2025-10-10T01:23:14.7019221Z * [new branch] gh/manuelcandales/9/head -> origin/gh/manuelcandales/9/head 2025-10-10T01:23:14.7021057Z * [new branch] gh/manuelcandales/9/orig -> origin/gh/manuelcandales/9/orig 2025-10-10T01:23:14.7023959Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-10-10T01:23:14.7027691Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-10-10T01:23:14.7029463Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-10-10T01:23:14.7031173Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-10-10T01:23:14.7033684Z * [new branch] gh/masnesral/236/base -> origin/gh/masnesral/236/base 2025-10-10T01:23:14.7035390Z * [new branch] gh/masnesral/236/head -> origin/gh/masnesral/236/head 2025-10-10T01:23:14.7037081Z * [new branch] gh/masnesral/236/orig -> origin/gh/masnesral/236/orig 2025-10-10T01:23:14.7039509Z * [new branch] gh/masnesral/237/base -> origin/gh/masnesral/237/base 2025-10-10T01:23:14.7041407Z * [new branch] gh/masnesral/237/head -> origin/gh/masnesral/237/head 2025-10-10T01:23:14.7043418Z * [new branch] gh/masnesral/237/orig -> origin/gh/masnesral/237/orig 2025-10-10T01:23:14.7045933Z * [new branch] gh/masnesral/238/base -> origin/gh/masnesral/238/base 2025-10-10T01:23:14.7047545Z * [new branch] gh/masnesral/238/head -> origin/gh/masnesral/238/head 2025-10-10T01:23:14.7049142Z * [new branch] gh/masnesral/238/orig -> origin/gh/masnesral/238/orig 2025-10-10T01:23:14.7051925Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-10-10T01:23:14.7053700Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-10-10T01:23:14.7055930Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-10-10T01:23:14.7057626Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-10-10T01:23:14.7059814Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-10-10T01:23:14.7061491Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-10-10T01:23:14.7063855Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-10-10T01:23:14.7065512Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-10-10T01:23:14.7067690Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-10-10T01:23:14.7069335Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-10-10T01:23:14.7071349Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-10-10T01:23:14.7073027Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-10-10T01:23:14.7075215Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-10-10T01:23:14.7076822Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-10-10T01:23:14.7079820Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-10-10T01:23:14.7081533Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-10-10T01:23:14.7084007Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-10-10T01:23:14.7085732Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-10-10T01:23:14.7088052Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-10-10T01:23:14.7089664Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-10-10T01:23:14.7091800Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-10-10T01:23:14.7093469Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-10-10T01:23:14.7095771Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-10-10T01:23:14.7099411Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-10-10T01:23:14.7101684Z * [new branch] gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base 2025-10-10T01:23:14.7103458Z * [new branch] gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head 2025-10-10T01:23:14.7105152Z * [new branch] gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig 2025-10-10T01:23:14.7107549Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-10-10T01:23:14.7109239Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-10-10T01:23:14.7110915Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-10-10T01:23:14.7113734Z * [new branch] gh/mikaylagawarecki/337/base -> origin/gh/mikaylagawarecki/337/base 2025-10-10T01:23:14.7115718Z * [new branch] gh/mikaylagawarecki/337/head -> origin/gh/mikaylagawarecki/337/head 2025-10-10T01:23:14.7117322Z * [new branch] gh/mikaylagawarecki/337/orig -> origin/gh/mikaylagawarecki/337/orig 2025-10-10T01:23:14.7119838Z * [new branch] gh/mikaylagawarecki/340/base -> origin/gh/mikaylagawarecki/340/base 2025-10-10T01:23:14.7121671Z * [new branch] gh/mikaylagawarecki/340/head -> origin/gh/mikaylagawarecki/340/head 2025-10-10T01:23:14.7123323Z * [new branch] gh/mikaylagawarecki/340/orig -> origin/gh/mikaylagawarecki/340/orig 2025-10-10T01:23:14.7125800Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-10-10T01:23:14.7127535Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-10-10T01:23:14.7129246Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-10-10T01:23:14.7131686Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-10-10T01:23:14.7133355Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-10-10T01:23:14.7135140Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-10-10T01:23:14.7137433Z * [new branch] gh/mikaylagawarecki/343/base -> origin/gh/mikaylagawarecki/343/base 2025-10-10T01:23:14.7139261Z * [new branch] gh/mikaylagawarecki/343/head -> origin/gh/mikaylagawarecki/343/head 2025-10-10T01:23:14.7141471Z * [new branch] gh/mikaylagawarecki/343/orig -> origin/gh/mikaylagawarecki/343/orig 2025-10-10T01:23:14.7143869Z * [new branch] gh/mikaylagawarecki/344/base -> origin/gh/mikaylagawarecki/344/base 2025-10-10T01:23:14.7145540Z * [new branch] gh/mikaylagawarecki/344/head -> origin/gh/mikaylagawarecki/344/head 2025-10-10T01:23:14.7147224Z * [new branch] gh/mikaylagawarecki/344/orig -> origin/gh/mikaylagawarecki/344/orig 2025-10-10T01:23:14.7149976Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-10-10T01:23:14.7151630Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-10-10T01:23:14.7153380Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-10-10T01:23:14.7155907Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-10-10T01:23:14.7157549Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-10-10T01:23:14.7159345Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-10-10T01:23:14.7161662Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-10-10T01:23:14.7163377Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-10-10T01:23:14.7165124Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-10-10T01:23:14.7167869Z * [new branch] gh/mikaylagawarecki/348/base -> origin/gh/mikaylagawarecki/348/base 2025-10-10T01:23:14.7169596Z * [new branch] gh/mikaylagawarecki/348/head -> origin/gh/mikaylagawarecki/348/head 2025-10-10T01:23:14.7171176Z * [new branch] gh/mikaylagawarecki/348/orig -> origin/gh/mikaylagawarecki/348/orig 2025-10-10T01:23:14.7173696Z * [new branch] gh/mikaylagawarecki/349/base -> origin/gh/mikaylagawarecki/349/base 2025-10-10T01:23:14.7175334Z * [new branch] gh/mikaylagawarecki/349/head -> origin/gh/mikaylagawarecki/349/head 2025-10-10T01:23:14.7177083Z * [new branch] gh/mikaylagawarecki/349/orig -> origin/gh/mikaylagawarecki/349/orig 2025-10-10T01:23:14.7179677Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-10-10T01:23:14.7181306Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-10-10T01:23:14.7182987Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-10-10T01:23:14.7185785Z * [new branch] gh/mlazos/18/base -> origin/gh/mlazos/18/base 2025-10-10T01:23:14.7187455Z * [new branch] gh/mlazos/18/head -> origin/gh/mlazos/18/head 2025-10-10T01:23:14.7189136Z * [new branch] gh/mlazos/18/orig -> origin/gh/mlazos/18/orig 2025-10-10T01:23:14.7191382Z * [new branch] gh/mlazos/19/base -> origin/gh/mlazos/19/base 2025-10-10T01:23:14.7193049Z * [new branch] gh/mlazos/19/head -> origin/gh/mlazos/19/head 2025-10-10T01:23:14.7194769Z * [new branch] gh/mlazos/19/orig -> origin/gh/mlazos/19/orig 2025-10-10T01:23:14.7197418Z * [new branch] gh/mlazos/20/base -> origin/gh/mlazos/20/base 2025-10-10T01:23:14.7199300Z * [new branch] gh/mlazos/20/head -> origin/gh/mlazos/20/head 2025-10-10T01:23:14.7201055Z * [new branch] gh/mlazos/20/orig -> origin/gh/mlazos/20/orig 2025-10-10T01:23:14.7203439Z * [new branch] gh/mlazos/21/base -> origin/gh/mlazos/21/base 2025-10-10T01:23:14.7205183Z * [new branch] gh/mlazos/21/head -> origin/gh/mlazos/21/head 2025-10-10T01:23:14.7206890Z * [new branch] gh/mlazos/21/orig -> origin/gh/mlazos/21/orig 2025-10-10T01:23:14.7209199Z * [new branch] gh/mlazos/22/base -> origin/gh/mlazos/22/base 2025-10-10T01:23:14.7210899Z * [new branch] gh/mlazos/22/head -> origin/gh/mlazos/22/head 2025-10-10T01:23:14.7212515Z * [new branch] gh/mlazos/22/orig -> origin/gh/mlazos/22/orig 2025-10-10T01:23:14.7215306Z * [new branch] gh/mlazos/23/base -> origin/gh/mlazos/23/base 2025-10-10T01:23:14.7217058Z * [new branch] gh/mlazos/23/head -> origin/gh/mlazos/23/head 2025-10-10T01:23:14.7218747Z * [new branch] gh/mlazos/23/orig -> origin/gh/mlazos/23/orig 2025-10-10T01:23:14.7221605Z * [new branch] gh/mlazos/24/base -> origin/gh/mlazos/24/base 2025-10-10T01:23:14.7223339Z * [new branch] gh/mlazos/24/head -> origin/gh/mlazos/24/head 2025-10-10T01:23:14.7225047Z * [new branch] gh/mlazos/24/orig -> origin/gh/mlazos/24/orig 2025-10-10T01:23:14.7227584Z * [new branch] gh/mlazos/25/base -> origin/gh/mlazos/25/base 2025-10-10T01:23:14.7229371Z * [new branch] gh/mlazos/25/head -> origin/gh/mlazos/25/head 2025-10-10T01:23:14.7231196Z * [new branch] gh/mlazos/25/orig -> origin/gh/mlazos/25/orig 2025-10-10T01:23:14.7233589Z * [new branch] gh/mlazos/26/base -> origin/gh/mlazos/26/base 2025-10-10T01:23:14.7235206Z * [new branch] gh/mlazos/26/head -> origin/gh/mlazos/26/head 2025-10-10T01:23:14.7236869Z * [new branch] gh/mlazos/26/orig -> origin/gh/mlazos/26/orig 2025-10-10T01:23:14.7239203Z * [new branch] gh/mlazos/27/base -> origin/gh/mlazos/27/base 2025-10-10T01:23:14.7240928Z * [new branch] gh/mlazos/27/head -> origin/gh/mlazos/27/head 2025-10-10T01:23:14.7242453Z * [new branch] gh/mlazos/27/orig -> origin/gh/mlazos/27/orig 2025-10-10T01:23:14.7244775Z * [new branch] gh/mlazos/28/base -> origin/gh/mlazos/28/base 2025-10-10T01:23:14.7246465Z * [new branch] gh/mlazos/28/head -> origin/gh/mlazos/28/head 2025-10-10T01:23:14.7248110Z * [new branch] gh/mlazos/28/orig -> origin/gh/mlazos/28/orig 2025-10-10T01:23:14.7250710Z * [new branch] gh/mlazos/29/base -> origin/gh/mlazos/29/base 2025-10-10T01:23:14.7252448Z * [new branch] gh/mlazos/29/head -> origin/gh/mlazos/29/head 2025-10-10T01:23:14.7254047Z * [new branch] gh/mlazos/29/orig -> origin/gh/mlazos/29/orig 2025-10-10T01:23:14.7256276Z * [new branch] gh/mlazos/30/base -> origin/gh/mlazos/30/base 2025-10-10T01:23:14.7258009Z * [new branch] gh/mlazos/30/head -> origin/gh/mlazos/30/head 2025-10-10T01:23:14.7259766Z * [new branch] gh/mlazos/30/orig -> origin/gh/mlazos/30/orig 2025-10-10T01:23:14.7262008Z * [new branch] gh/mlazos/31/base -> origin/gh/mlazos/31/base 2025-10-10T01:23:14.7263697Z * [new branch] gh/mlazos/31/head -> origin/gh/mlazos/31/head 2025-10-10T01:23:14.7265380Z * [new branch] gh/mlazos/31/orig -> origin/gh/mlazos/31/orig 2025-10-10T01:23:14.7267741Z * [new branch] gh/mlazos/32/base -> origin/gh/mlazos/32/base 2025-10-10T01:23:14.7269400Z * [new branch] gh/mlazos/32/head -> origin/gh/mlazos/32/head 2025-10-10T01:23:14.7271107Z * [new branch] gh/mlazos/32/orig -> origin/gh/mlazos/32/orig 2025-10-10T01:23:14.7273523Z * [new branch] gh/mlazos/33/base -> origin/gh/mlazos/33/base 2025-10-10T01:23:14.7275353Z * [new branch] gh/mlazos/33/head -> origin/gh/mlazos/33/head 2025-10-10T01:23:14.7277011Z * [new branch] gh/mlazos/33/orig -> origin/gh/mlazos/33/orig 2025-10-10T01:23:14.7279339Z * [new branch] gh/mlazos/34/base -> origin/gh/mlazos/34/base 2025-10-10T01:23:14.7281040Z * [new branch] gh/mlazos/34/head -> origin/gh/mlazos/34/head 2025-10-10T01:23:14.7282643Z * [new branch] gh/mlazos/34/orig -> origin/gh/mlazos/34/orig 2025-10-10T01:23:14.7284821Z * [new branch] gh/mlazos/35/base -> origin/gh/mlazos/35/base 2025-10-10T01:23:14.7286521Z * [new branch] gh/mlazos/35/head -> origin/gh/mlazos/35/head 2025-10-10T01:23:14.7288220Z * [new branch] gh/mlazos/35/orig -> origin/gh/mlazos/35/orig 2025-10-10T01:23:14.7291517Z * [new branch] gh/mlazos/36/base -> origin/gh/mlazos/36/base 2025-10-10T01:23:14.7293229Z * [new branch] gh/mlazos/36/head -> origin/gh/mlazos/36/head 2025-10-10T01:23:14.7294850Z * [new branch] gh/mlazos/36/orig -> origin/gh/mlazos/36/orig 2025-10-10T01:23:14.7299354Z * [new branch] gh/mlazos/37/base -> origin/gh/mlazos/37/base 2025-10-10T01:23:14.7301078Z * [new branch] gh/mlazos/37/head -> origin/gh/mlazos/37/head 2025-10-10T01:23:14.7302851Z * [new branch] gh/mlazos/37/orig -> origin/gh/mlazos/37/orig 2025-10-10T01:23:14.7305614Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-10-10T01:23:14.7307473Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-10-10T01:23:14.7310322Z * [new branch] gh/muchulee8/62/base -> origin/gh/muchulee8/62/base 2025-10-10T01:23:14.7312032Z * [new branch] gh/muchulee8/62/head -> origin/gh/muchulee8/62/head 2025-10-10T01:23:14.7313897Z * [new branch] gh/muchulee8/62/orig -> origin/gh/muchulee8/62/orig 2025-10-10T01:23:14.7316333Z * [new branch] gh/muchulee8/64/base -> origin/gh/muchulee8/64/base 2025-10-10T01:23:14.7318086Z * [new branch] gh/muchulee8/64/head -> origin/gh/muchulee8/64/head 2025-10-10T01:23:14.7319954Z * [new branch] gh/muchulee8/64/orig -> origin/gh/muchulee8/64/orig 2025-10-10T01:23:14.7322399Z * [new branch] gh/muchulee8/65/base -> origin/gh/muchulee8/65/base 2025-10-10T01:23:14.7324229Z * [new branch] gh/muchulee8/65/head -> origin/gh/muchulee8/65/head 2025-10-10T01:23:14.7325959Z * [new branch] gh/muchulee8/65/orig -> origin/gh/muchulee8/65/orig 2025-10-10T01:23:14.7328214Z * [new branch] gh/muchulee8/66/base -> origin/gh/muchulee8/66/base 2025-10-10T01:23:14.7329923Z * [new branch] gh/muchulee8/66/head -> origin/gh/muchulee8/66/head 2025-10-10T01:23:14.7331638Z * [new branch] gh/muchulee8/66/orig -> origin/gh/muchulee8/66/orig 2025-10-10T01:23:14.7333984Z * [new branch] gh/muchulee8/67/base -> origin/gh/muchulee8/67/base 2025-10-10T01:23:14.7335752Z * [new branch] gh/muchulee8/67/head -> origin/gh/muchulee8/67/head 2025-10-10T01:23:14.7337497Z * [new branch] gh/muchulee8/67/orig -> origin/gh/muchulee8/67/orig 2025-10-10T01:23:14.7340321Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-10-10T01:23:14.7342144Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-10-10T01:23:14.7343994Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-10-10T01:23:14.7346316Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-10-10T01:23:14.7347993Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-10-10T01:23:14.7349830Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-10-10T01:23:14.7352031Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-10-10T01:23:14.7353726Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-10-10T01:23:14.7355512Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-10-10T01:23:14.7357789Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-10-10T01:23:14.7359531Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-10-10T01:23:14.7361403Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-10-10T01:23:14.7363821Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-10-10T01:23:14.7365994Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-10-10T01:23:14.7367833Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-10-10T01:23:14.7370155Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-10-10T01:23:14.7371940Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-10-10T01:23:14.7373682Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-10-10T01:23:14.7376380Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-10-10T01:23:14.7378078Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-10-10T01:23:14.7379756Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-10-10T01:23:14.7382171Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-10-10T01:23:14.7383889Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-10-10T01:23:14.7385635Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-10-10T01:23:14.7388930Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-10-10T01:23:14.7390749Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-10-10T01:23:14.7392769Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-10-10T01:23:14.7395016Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-10-10T01:23:14.7396861Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-10-10T01:23:14.7398731Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-10-10T01:23:14.7401169Z * [new branch] gh/nikitaved/3/base -> origin/gh/nikitaved/3/base 2025-10-10T01:23:14.7402805Z * [new branch] gh/nikitaved/3/head -> origin/gh/nikitaved/3/head 2025-10-10T01:23:14.7404478Z * [new branch] gh/nikitaved/3/orig -> origin/gh/nikitaved/3/orig 2025-10-10T01:23:14.7407232Z * [new branch] gh/oulgen/35/base -> origin/gh/oulgen/35/base 2025-10-10T01:23:14.7408896Z * [new branch] gh/oulgen/35/head -> origin/gh/oulgen/35/head 2025-10-10T01:23:14.7410641Z * [new branch] gh/oulgen/35/orig -> origin/gh/oulgen/35/orig 2025-10-10T01:23:14.7413170Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-10-10T01:23:14.7416124Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-10-10T01:23:14.7417762Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-10-10T01:23:14.7419619Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-10-10T01:23:14.7422020Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-10-10T01:23:14.7423772Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-10-10T01:23:14.7425567Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-10-10T01:23:14.7427855Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-10-10T01:23:14.7429585Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-10-10T01:23:14.7431200Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-10-10T01:23:14.7433681Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-10-10T01:23:14.7435314Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-10-10T01:23:14.7437086Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-10-10T01:23:14.7439483Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-10-10T01:23:14.7441182Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-10-10T01:23:14.7442836Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-10-10T01:23:14.7445109Z * [new branch] gh/pearu/113/base -> origin/gh/pearu/113/base 2025-10-10T01:23:14.7446781Z * [new branch] gh/pearu/113/head -> origin/gh/pearu/113/head 2025-10-10T01:23:14.7448459Z * [new branch] gh/pearu/113/orig -> origin/gh/pearu/113/orig 2025-10-10T01:23:14.7450766Z * [new branch] gh/pearu/114/base -> origin/gh/pearu/114/base 2025-10-10T01:23:14.7452505Z * [new branch] gh/pearu/114/head -> origin/gh/pearu/114/head 2025-10-10T01:23:14.7454157Z * [new branch] gh/pearu/114/orig -> origin/gh/pearu/114/orig 2025-10-10T01:23:14.7456466Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-10-10T01:23:14.7458253Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-10-10T01:23:14.7459935Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-10-10T01:23:14.7462122Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-10-10T01:23:14.7464006Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-10-10T01:23:14.7465753Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-10-10T01:23:14.7468009Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-10-10T01:23:14.7469663Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-10-10T01:23:14.7471454Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-10-10T01:23:14.7481488Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-10-10T01:23:14.7481898Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-10-10T01:23:14.7482098Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-10-10T01:23:14.7482275Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-10-10T01:23:14.7482463Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-10-10T01:23:14.7483391Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-10-10T01:23:14.7485910Z * [new branch] gh/pearu/120/base -> origin/gh/pearu/120/base 2025-10-10T01:23:14.7487613Z * [new branch] gh/pearu/120/head -> origin/gh/pearu/120/head 2025-10-10T01:23:14.7489316Z * [new branch] gh/pearu/120/orig -> origin/gh/pearu/120/orig 2025-10-10T01:23:14.7491500Z * [new branch] gh/pearu/121/base -> origin/gh/pearu/121/base 2025-10-10T01:23:14.7493180Z * [new branch] gh/pearu/121/head -> origin/gh/pearu/121/head 2025-10-10T01:23:14.7494864Z * [new branch] gh/pearu/121/orig -> origin/gh/pearu/121/orig 2025-10-10T01:23:14.7497312Z * [new branch] gh/pearu/122/base -> origin/gh/pearu/122/base 2025-10-10T01:23:14.7499089Z * [new branch] gh/pearu/122/head -> origin/gh/pearu/122/head 2025-10-10T01:23:14.7500845Z * [new branch] gh/pearu/122/orig -> origin/gh/pearu/122/orig 2025-10-10T01:23:14.7503169Z * [new branch] gh/pearu/123/base -> origin/gh/pearu/123/base 2025-10-10T01:23:14.7504941Z * [new branch] gh/pearu/123/head -> origin/gh/pearu/123/head 2025-10-10T01:23:14.7506643Z * [new branch] gh/pearu/123/orig -> origin/gh/pearu/123/orig 2025-10-10T01:23:14.7509538Z * [new branch] gh/pearu/124/base -> origin/gh/pearu/124/base 2025-10-10T01:23:14.7511121Z * [new branch] gh/pearu/124/head -> origin/gh/pearu/124/head 2025-10-10T01:23:14.7513085Z * [new branch] gh/pearu/124/orig -> origin/gh/pearu/124/orig 2025-10-10T01:23:14.7515384Z * [new branch] gh/pearu/125/base -> origin/gh/pearu/125/base 2025-10-10T01:23:14.7516999Z * [new branch] gh/pearu/125/head -> origin/gh/pearu/125/head 2025-10-10T01:23:14.7518657Z * [new branch] gh/pearu/125/orig -> origin/gh/pearu/125/orig 2025-10-10T01:23:14.7521183Z * [new branch] gh/pearu/126/base -> origin/gh/pearu/126/base 2025-10-10T01:23:14.7522829Z * [new branch] gh/pearu/126/head -> origin/gh/pearu/126/head 2025-10-10T01:23:14.7524448Z * [new branch] gh/pearu/126/orig -> origin/gh/pearu/126/orig 2025-10-10T01:23:14.7526780Z * [new branch] gh/pearu/127/base -> origin/gh/pearu/127/base 2025-10-10T01:23:14.7528512Z * [new branch] gh/pearu/127/head -> origin/gh/pearu/127/head 2025-10-10T01:23:14.7530126Z * [new branch] gh/pearu/127/orig -> origin/gh/pearu/127/orig 2025-10-10T01:23:14.7532967Z * [new branch] gh/pearu/128/base -> origin/gh/pearu/128/base 2025-10-10T01:23:14.7534824Z * [new branch] gh/pearu/128/head -> origin/gh/pearu/128/head 2025-10-10T01:23:14.7536356Z * [new branch] gh/pearu/128/orig -> origin/gh/pearu/128/orig 2025-10-10T01:23:14.7538837Z * [new branch] gh/pearu/129/base -> origin/gh/pearu/129/base 2025-10-10T01:23:14.7540523Z * [new branch] gh/pearu/129/head -> origin/gh/pearu/129/head 2025-10-10T01:23:14.7542199Z * [new branch] gh/pearu/129/orig -> origin/gh/pearu/129/orig 2025-10-10T01:23:14.7544880Z * [new branch] gh/pearu/130/base -> origin/gh/pearu/130/base 2025-10-10T01:23:14.7546590Z * [new branch] gh/pearu/130/head -> origin/gh/pearu/130/head 2025-10-10T01:23:14.7548278Z * [new branch] gh/pearu/130/orig -> origin/gh/pearu/130/orig 2025-10-10T01:23:14.7550641Z * [new branch] gh/pearu/131/base -> origin/gh/pearu/131/base 2025-10-10T01:23:14.7552454Z * [new branch] gh/pearu/131/head -> origin/gh/pearu/131/head 2025-10-10T01:23:14.7554158Z * [new branch] gh/pearu/131/orig -> origin/gh/pearu/131/orig 2025-10-10T01:23:14.7556426Z * [new branch] gh/pearu/132/base -> origin/gh/pearu/132/base 2025-10-10T01:23:14.7558006Z * [new branch] gh/pearu/132/head -> origin/gh/pearu/132/head 2025-10-10T01:23:14.7559773Z * [new branch] gh/pearu/132/orig -> origin/gh/pearu/132/orig 2025-10-10T01:23:14.7561989Z * [new branch] gh/pearu/133/base -> origin/gh/pearu/133/base 2025-10-10T01:23:14.7563596Z * [new branch] gh/pearu/133/head -> origin/gh/pearu/133/head 2025-10-10T01:23:14.7565196Z * [new branch] gh/pearu/133/orig -> origin/gh/pearu/133/orig 2025-10-10T01:23:14.7568357Z * [new branch] gh/pearu/134/base -> origin/gh/pearu/134/base 2025-10-10T01:23:14.7569861Z * [new branch] gh/pearu/134/head -> origin/gh/pearu/134/head 2025-10-10T01:23:14.7571233Z * [new branch] gh/pearu/134/orig -> origin/gh/pearu/134/orig 2025-10-10T01:23:14.7573925Z * [new branch] gh/pearu/135/base -> origin/gh/pearu/135/base 2025-10-10T01:23:14.7575674Z * [new branch] gh/pearu/135/head -> origin/gh/pearu/135/head 2025-10-10T01:23:14.7577337Z * [new branch] gh/pearu/135/orig -> origin/gh/pearu/135/orig 2025-10-10T01:23:14.7579667Z * [new branch] gh/pearu/136/base -> origin/gh/pearu/136/base 2025-10-10T01:23:14.7581309Z * [new branch] gh/pearu/136/head -> origin/gh/pearu/136/head 2025-10-10T01:23:14.7582966Z * [new branch] gh/pearu/136/orig -> origin/gh/pearu/136/orig 2025-10-10T01:23:14.7585320Z * [new branch] gh/pearu/137/base -> origin/gh/pearu/137/base 2025-10-10T01:23:14.7587038Z * [new branch] gh/pearu/137/head -> origin/gh/pearu/137/head 2025-10-10T01:23:14.7588777Z * [new branch] gh/pearu/137/orig -> origin/gh/pearu/137/orig 2025-10-10T01:23:14.7591252Z * [new branch] gh/pearu/138/base -> origin/gh/pearu/138/base 2025-10-10T01:23:14.7593070Z * [new branch] gh/pearu/138/head -> origin/gh/pearu/138/head 2025-10-10T01:23:14.7594686Z * [new branch] gh/pearu/138/orig -> origin/gh/pearu/138/orig 2025-10-10T01:23:14.7597354Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-10-10T01:23:14.7599041Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-10-10T01:23:14.7600908Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-10-10T01:23:14.7604114Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-10-10T01:23:14.7606254Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-10-10T01:23:14.7607739Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-10-10T01:23:14.7610277Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-10-10T01:23:14.7611984Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-10-10T01:23:14.7613674Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-10-10T01:23:14.7616430Z * [new branch] gh/pianpwk/1/base -> origin/gh/pianpwk/1/base 2025-10-10T01:23:14.7618115Z * [new branch] gh/pianpwk/1/head -> origin/gh/pianpwk/1/head 2025-10-10T01:23:14.7619841Z * [new branch] gh/pianpwk/1/orig -> origin/gh/pianpwk/1/orig 2025-10-10T01:23:14.7622136Z * [new branch] gh/pianpwk/2/base -> origin/gh/pianpwk/2/base 2025-10-10T01:23:14.7623806Z * [new branch] gh/pianpwk/2/head -> origin/gh/pianpwk/2/head 2025-10-10T01:23:14.7625466Z * [new branch] gh/pianpwk/2/orig -> origin/gh/pianpwk/2/orig 2025-10-10T01:23:14.7627884Z * [new branch] gh/pianpwk/3/base -> origin/gh/pianpwk/3/base 2025-10-10T01:23:14.7629515Z * [new branch] gh/pianpwk/3/head -> origin/gh/pianpwk/3/head 2025-10-10T01:23:14.7631340Z * [new branch] gh/pianpwk/3/orig -> origin/gh/pianpwk/3/orig 2025-10-10T01:23:14.7633629Z * [new branch] gh/pianpwk/4/base -> origin/gh/pianpwk/4/base 2025-10-10T01:23:14.7635402Z * [new branch] gh/pianpwk/4/head -> origin/gh/pianpwk/4/head 2025-10-10T01:23:14.7637455Z * [new branch] gh/pianpwk/4/orig -> origin/gh/pianpwk/4/orig 2025-10-10T01:23:14.7639838Z * [new branch] gh/pianpwk/5/base -> origin/gh/pianpwk/5/base 2025-10-10T01:23:14.7641472Z * [new branch] gh/pianpwk/5/head -> origin/gh/pianpwk/5/head 2025-10-10T01:23:14.7643348Z * [new branch] gh/pianpwk/5/orig -> origin/gh/pianpwk/5/orig 2025-10-10T01:23:14.7645641Z * [new branch] gh/pianpwk/6/base -> origin/gh/pianpwk/6/base 2025-10-10T01:23:14.7647305Z * [new branch] gh/pianpwk/6/head -> origin/gh/pianpwk/6/head 2025-10-10T01:23:14.7648959Z * [new branch] gh/pianpwk/6/orig -> origin/gh/pianpwk/6/orig 2025-10-10T01:23:14.7651227Z * [new branch] gh/pianpwk/7/base -> origin/gh/pianpwk/7/base 2025-10-10T01:23:14.7652849Z * [new branch] gh/pianpwk/7/head -> origin/gh/pianpwk/7/head 2025-10-10T01:23:14.7654578Z * [new branch] gh/pianpwk/7/orig -> origin/gh/pianpwk/7/orig 2025-10-10T01:23:14.7657322Z * [new branch] gh/pianpwk/8/base -> origin/gh/pianpwk/8/base 2025-10-10T01:23:14.7659025Z * [new branch] gh/pianpwk/8/head -> origin/gh/pianpwk/8/head 2025-10-10T01:23:14.7660697Z * [new branch] gh/pianpwk/8/orig -> origin/gh/pianpwk/8/orig 2025-10-10T01:23:14.7662978Z * [new branch] gh/raymo/refresh-script -> origin/gh/raymo/refresh-script 2025-10-10T01:23:14.7665884Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-10-10T01:23:14.7667822Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-10-10T01:23:14.7670128Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-10-10T01:23:14.7672071Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-10-10T01:23:14.7673915Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-10-10T01:23:14.7676215Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-10-10T01:23:14.7678042Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-10-10T01:23:14.7679712Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-10-10T01:23:14.7681976Z * [new branch] gh/rec/162/base -> origin/gh/rec/162/base 2025-10-10T01:23:14.7683671Z * [new branch] gh/rec/162/head -> origin/gh/rec/162/head 2025-10-10T01:23:14.7685343Z * [new branch] gh/rec/162/orig -> origin/gh/rec/162/orig 2025-10-10T01:23:14.7687590Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-10-10T01:23:14.7689370Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-10-10T01:23:14.7691297Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-10-10T01:23:14.7694041Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-10-10T01:23:14.7695809Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-10-10T01:23:14.7699143Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-10-10T01:23:14.7702056Z * [new branch] gh/robert-hardwick/1/base -> origin/gh/robert-hardwick/1/base 2025-10-10T01:23:14.7703731Z * [new branch] gh/robert-hardwick/1/head -> origin/gh/robert-hardwick/1/head 2025-10-10T01:23:14.7705446Z * [new branch] gh/robert-hardwick/1/orig -> origin/gh/robert-hardwick/1/orig 2025-10-10T01:23:14.7707775Z * [new branch] gh/robert-hardwick/2/base -> origin/gh/robert-hardwick/2/base 2025-10-10T01:23:14.7709517Z * [new branch] gh/robert-hardwick/2/head -> origin/gh/robert-hardwick/2/head 2025-10-10T01:23:14.7711251Z * [new branch] gh/robert-hardwick/2/orig -> origin/gh/robert-hardwick/2/orig 2025-10-10T01:23:14.7713687Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-10-10T01:23:14.7715392Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-10-10T01:23:14.7717099Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-10-10T01:23:14.7719393Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-10-10T01:23:14.7721103Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-10-10T01:23:14.7722779Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-10-10T01:23:14.7725553Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-10-10T01:23:14.7727286Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-10-10T01:23:14.7729618Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-10-10T01:23:14.7731395Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-10-10T01:23:14.7733209Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-10-10T01:23:14.7735590Z * [new branch] gh/rtimpe/15/base -> origin/gh/rtimpe/15/base 2025-10-10T01:23:14.7737272Z * [new branch] gh/rtimpe/15/head -> origin/gh/rtimpe/15/head 2025-10-10T01:23:14.7739027Z * [new branch] gh/rtimpe/15/orig -> origin/gh/rtimpe/15/orig 2025-10-10T01:23:14.7741206Z * [new branch] gh/rtimpe/16/base -> origin/gh/rtimpe/16/base 2025-10-10T01:23:14.7742884Z * [new branch] gh/rtimpe/16/head -> origin/gh/rtimpe/16/head 2025-10-10T01:23:14.7744643Z * [new branch] gh/rtimpe/16/orig -> origin/gh/rtimpe/16/orig 2025-10-10T01:23:14.7746934Z * [new branch] gh/rtimpe/17/base -> origin/gh/rtimpe/17/base 2025-10-10T01:23:14.7748847Z * [new branch] gh/rtimpe/17/head -> origin/gh/rtimpe/17/head 2025-10-10T01:23:14.7750388Z * [new branch] gh/rtimpe/17/orig -> origin/gh/rtimpe/17/orig 2025-10-10T01:23:14.7752647Z * [new branch] gh/rtimpe/18/base -> origin/gh/rtimpe/18/base 2025-10-10T01:23:14.7754455Z * [new branch] gh/rtimpe/18/head -> origin/gh/rtimpe/18/head 2025-10-10T01:23:14.7756175Z * [new branch] gh/rtimpe/18/orig -> origin/gh/rtimpe/18/orig 2025-10-10T01:23:14.7758517Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-10-10T01:23:14.7760297Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-10-10T01:23:14.7762879Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-10-10T01:23:14.7764451Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-10-10T01:23:14.7766661Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-10-10T01:23:14.7768353Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-10-10T01:23:14.7771258Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-10-10T01:23:14.7772855Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-10-10T01:23:14.7774550Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-10-10T01:23:14.7776879Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-10-10T01:23:14.7778576Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-10-10T01:23:14.7780355Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-10-10T01:23:14.7782681Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-10-10T01:23:14.7784385Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-10-10T01:23:14.7786067Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-10-10T01:23:14.7788312Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-10-10T01:23:14.7790022Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-10-10T01:23:14.7791762Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-10-10T01:23:14.7794007Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-10-10T01:23:14.7795721Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-10-10T01:23:14.7797626Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-10-10T01:23:14.7799959Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-10-10T01:23:14.7801710Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-10-10T01:23:14.7803633Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-10-10T01:23:14.7805879Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-10-10T01:23:14.7807543Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-10-10T01:23:14.7809237Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-10-10T01:23:14.7812023Z * [new branch] gh/sarckk/2/base -> origin/gh/sarckk/2/base 2025-10-10T01:23:14.7813814Z * [new branch] gh/sarckk/2/head -> origin/gh/sarckk/2/head 2025-10-10T01:23:14.7815574Z * [new branch] gh/sarckk/2/orig -> origin/gh/sarckk/2/orig 2025-10-10T01:23:14.7818512Z * [new branch] gh/seemethere/35/base -> origin/gh/seemethere/35/base 2025-10-10T01:23:14.7820091Z * [new branch] gh/seemethere/35/head -> origin/gh/seemethere/35/head 2025-10-10T01:23:14.7821780Z * [new branch] gh/seemethere/35/orig -> origin/gh/seemethere/35/orig 2025-10-10T01:23:14.7824207Z * [new branch] gh/seemethere/37/base -> origin/gh/seemethere/37/base 2025-10-10T01:23:14.7825874Z * [new branch] gh/seemethere/37/head -> origin/gh/seemethere/37/head 2025-10-10T01:23:14.7827575Z * [new branch] gh/seemethere/37/orig -> origin/gh/seemethere/37/orig 2025-10-10T01:23:14.7829855Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-10-10T01:23:14.7831610Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-10-10T01:23:14.7833432Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-10-10T01:23:14.7835679Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-10-10T01:23:14.7837378Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-10-10T01:23:14.7839250Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-10-10T01:23:14.7841391Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-10-10T01:23:14.7843549Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-10-10T01:23:14.7845229Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-10-10T01:23:14.7847646Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-10-10T01:23:14.7849278Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-10-10T01:23:14.7850972Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-10-10T01:23:14.7853373Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-10-10T01:23:14.7855111Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-10-10T01:23:14.7856854Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-10-10T01:23:14.7859579Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-10-10T01:23:14.7861204Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-10-10T01:23:14.7862993Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-10-10T01:23:14.7865198Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-10-10T01:23:14.7866870Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-10-10T01:23:14.7868611Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-10-10T01:23:14.7871144Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-10-10T01:23:14.7873194Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-10-10T01:23:14.7875529Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-10-10T01:23:14.7877750Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-10-10T01:23:14.7879527Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-10-10T01:23:14.7881280Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-10-10T01:23:14.7883701Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-10-10T01:23:14.7885394Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-10-10T01:23:14.7887199Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-10-10T01:23:14.7889380Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-10-10T01:23:14.7891032Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-10-10T01:23:14.7892737Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-10-10T01:23:14.7895116Z * [new branch] gh/seemethere/64/base -> origin/gh/seemethere/64/base 2025-10-10T01:23:14.7896807Z * [new branch] gh/seemethere/64/head -> origin/gh/seemethere/64/head 2025-10-10T01:23:14.7898724Z * [new branch] gh/seemethere/64/orig -> origin/gh/seemethere/64/orig 2025-10-10T01:23:14.7901114Z * [new branch] gh/seemethere/65/base -> origin/gh/seemethere/65/base 2025-10-10T01:23:14.7902988Z * [new branch] gh/seemethere/65/head -> origin/gh/seemethere/65/head 2025-10-10T01:23:14.7904690Z * [new branch] gh/seemethere/65/orig -> origin/gh/seemethere/65/orig 2025-10-10T01:23:14.7906963Z * [new branch] gh/seemethere/66/base -> origin/gh/seemethere/66/base 2025-10-10T01:23:14.7908698Z * [new branch] gh/seemethere/66/head -> origin/gh/seemethere/66/head 2025-10-10T01:23:14.7910359Z * [new branch] gh/seemethere/66/orig -> origin/gh/seemethere/66/orig 2025-10-10T01:23:14.7912685Z * [new branch] gh/seemethere/67/base -> origin/gh/seemethere/67/base 2025-10-10T01:23:14.7914401Z * [new branch] gh/seemethere/67/head -> origin/gh/seemethere/67/head 2025-10-10T01:23:14.7916186Z * [new branch] gh/seemethere/67/orig -> origin/gh/seemethere/67/orig 2025-10-10T01:23:14.7919021Z * [new branch] gh/seemethere/68/base -> origin/gh/seemethere/68/base 2025-10-10T01:23:14.7920872Z * [new branch] gh/seemethere/68/head -> origin/gh/seemethere/68/head 2025-10-10T01:23:14.7922492Z * [new branch] gh/seemethere/68/orig -> origin/gh/seemethere/68/orig 2025-10-10T01:23:14.7925294Z * [new branch] gh/seemethere/69/base -> origin/gh/seemethere/69/base 2025-10-10T01:23:14.7926952Z * [new branch] gh/seemethere/69/head -> origin/gh/seemethere/69/head 2025-10-10T01:23:14.7928757Z * [new branch] gh/seemethere/69/orig -> origin/gh/seemethere/69/orig 2025-10-10T01:23:14.7931082Z * [new branch] gh/seemethere/70/base -> origin/gh/seemethere/70/base 2025-10-10T01:23:14.7932797Z * [new branch] gh/seemethere/70/head -> origin/gh/seemethere/70/head 2025-10-10T01:23:14.7934582Z * [new branch] gh/seemethere/70/orig -> origin/gh/seemethere/70/orig 2025-10-10T01:23:14.7936842Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-10-10T01:23:14.7938510Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-10-10T01:23:14.7940218Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-10-10T01:23:14.7943198Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-10-10T01:23:14.7945001Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-10-10T01:23:14.7950014Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-10-10T01:23:14.7953006Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-10-10T01:23:14.7954881Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-10-10T01:23:14.7956617Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-10-10T01:23:14.7959090Z * [new branch] gh/shunting314/211/base -> origin/gh/shunting314/211/base 2025-10-10T01:23:14.7960901Z * [new branch] gh/shunting314/211/head -> origin/gh/shunting314/211/head 2025-10-10T01:23:14.7962678Z * [new branch] gh/shunting314/211/orig -> origin/gh/shunting314/211/orig 2025-10-10T01:23:14.7965183Z * [new branch] gh/shunting314/212/base -> origin/gh/shunting314/212/base 2025-10-10T01:23:14.7966882Z * [new branch] gh/shunting314/212/head -> origin/gh/shunting314/212/head 2025-10-10T01:23:14.7968585Z * [new branch] gh/shunting314/212/orig -> origin/gh/shunting314/212/orig 2025-10-10T01:23:14.7971169Z * [new branch] gh/shunting314/213/base -> origin/gh/shunting314/213/base 2025-10-10T01:23:14.7972907Z * [new branch] gh/shunting314/213/head -> origin/gh/shunting314/213/head 2025-10-10T01:23:14.7974603Z * [new branch] gh/shunting314/213/orig -> origin/gh/shunting314/213/orig 2025-10-10T01:23:14.7977018Z * [new branch] gh/shunting314/215/base -> origin/gh/shunting314/215/base 2025-10-10T01:23:14.7978641Z * [new branch] gh/shunting314/215/head -> origin/gh/shunting314/215/head 2025-10-10T01:23:14.7980342Z * [new branch] gh/shunting314/215/orig -> origin/gh/shunting314/215/orig 2025-10-10T01:23:14.7982708Z * [new branch] gh/shunting314/216/base -> origin/gh/shunting314/216/base 2025-10-10T01:23:14.7984471Z * [new branch] gh/shunting314/216/head -> origin/gh/shunting314/216/head 2025-10-10T01:23:14.7986169Z * [new branch] gh/shunting314/216/orig -> origin/gh/shunting314/216/orig 2025-10-10T01:23:14.7988536Z * [new branch] gh/shunting314/217/base -> origin/gh/shunting314/217/base 2025-10-10T01:23:14.7990160Z * [new branch] gh/shunting314/217/head -> origin/gh/shunting314/217/head 2025-10-10T01:23:14.7991929Z * [new branch] gh/shunting314/217/orig -> origin/gh/shunting314/217/orig 2025-10-10T01:23:14.7994368Z * [new branch] gh/shunting314/218/base -> origin/gh/shunting314/218/base 2025-10-10T01:23:14.7996216Z * [new branch] gh/shunting314/218/head -> origin/gh/shunting314/218/head 2025-10-10T01:23:14.7998358Z * [new branch] gh/shunting314/218/orig -> origin/gh/shunting314/218/orig 2025-10-10T01:23:14.8000659Z * [new branch] gh/shunting314/219/base -> origin/gh/shunting314/219/base 2025-10-10T01:23:14.8002341Z * [new branch] gh/shunting314/219/head -> origin/gh/shunting314/219/head 2025-10-10T01:23:14.8004076Z * [new branch] gh/shunting314/219/orig -> origin/gh/shunting314/219/orig 2025-10-10T01:23:14.8006920Z * [new branch] gh/shunting314/223/base -> origin/gh/shunting314/223/base 2025-10-10T01:23:14.8008585Z * [new branch] gh/shunting314/223/head -> origin/gh/shunting314/223/head 2025-10-10T01:23:14.8010316Z * [new branch] gh/shunting314/223/orig -> origin/gh/shunting314/223/orig 2025-10-10T01:23:14.8012544Z * [new branch] gh/shunting314/224/base -> origin/gh/shunting314/224/base 2025-10-10T01:23:14.8014249Z * [new branch] gh/shunting314/224/head -> origin/gh/shunting314/224/head 2025-10-10T01:23:14.8015947Z * [new branch] gh/shunting314/224/orig -> origin/gh/shunting314/224/orig 2025-10-10T01:23:14.8018346Z * [new branch] gh/shunting314/225/base -> origin/gh/shunting314/225/base 2025-10-10T01:23:14.8020001Z * [new branch] gh/shunting314/225/head -> origin/gh/shunting314/225/head 2025-10-10T01:23:14.8021726Z * [new branch] gh/shunting314/225/orig -> origin/gh/shunting314/225/orig 2025-10-10T01:23:14.8024226Z * [new branch] gh/shunting314/226/base -> origin/gh/shunting314/226/base 2025-10-10T01:23:14.8026048Z * [new branch] gh/shunting314/226/head -> origin/gh/shunting314/226/head 2025-10-10T01:23:14.8027592Z * [new branch] gh/shunting314/226/orig -> origin/gh/shunting314/226/orig 2025-10-10T01:23:14.8029997Z * [new branch] gh/shunting314/227/base -> origin/gh/shunting314/227/base 2025-10-10T01:23:14.8031688Z * [new branch] gh/shunting314/227/head -> origin/gh/shunting314/227/head 2025-10-10T01:23:14.8033498Z * [new branch] gh/shunting314/227/orig -> origin/gh/shunting314/227/orig 2025-10-10T01:23:14.8035849Z * [new branch] gh/shunting314/228/base -> origin/gh/shunting314/228/base 2025-10-10T01:23:14.8037535Z * [new branch] gh/shunting314/228/head -> origin/gh/shunting314/228/head 2025-10-10T01:23:14.8039810Z * [new branch] gh/shunting314/228/orig -> origin/gh/shunting314/228/orig 2025-10-10T01:23:14.8043058Z * [new branch] gh/shunting314/229/base -> origin/gh/shunting314/229/base 2025-10-10T01:23:14.8044757Z * [new branch] gh/shunting314/229/head -> origin/gh/shunting314/229/head 2025-10-10T01:23:14.8046448Z * [new branch] gh/shunting314/229/orig -> origin/gh/shunting314/229/orig 2025-10-10T01:23:14.8048925Z * [new branch] gh/shunting314/230/base -> origin/gh/shunting314/230/base 2025-10-10T01:23:14.8050671Z * [new branch] gh/shunting314/230/head -> origin/gh/shunting314/230/head 2025-10-10T01:23:14.8052730Z * [new branch] gh/shunting314/230/orig -> origin/gh/shunting314/230/orig 2025-10-10T01:23:14.8055054Z * [new branch] gh/shunting314/231/base -> origin/gh/shunting314/231/base 2025-10-10T01:23:14.8056784Z * [new branch] gh/shunting314/231/head -> origin/gh/shunting314/231/head 2025-10-10T01:23:14.8058492Z * [new branch] gh/shunting314/231/orig -> origin/gh/shunting314/231/orig 2025-10-10T01:23:14.8061150Z * [new branch] gh/shunting314/232/base -> origin/gh/shunting314/232/base 2025-10-10T01:23:14.8062857Z * [new branch] gh/shunting314/232/head -> origin/gh/shunting314/232/head 2025-10-10T01:23:14.8064488Z * [new branch] gh/shunting314/232/orig -> origin/gh/shunting314/232/orig 2025-10-10T01:23:14.8066822Z * [new branch] gh/shunting314/233/base -> origin/gh/shunting314/233/base 2025-10-10T01:23:14.8068566Z * [new branch] gh/shunting314/233/head -> origin/gh/shunting314/233/head 2025-10-10T01:23:14.8070213Z * [new branch] gh/shunting314/233/orig -> origin/gh/shunting314/233/orig 2025-10-10T01:23:14.8072451Z * [new branch] gh/shunting314/234/base -> origin/gh/shunting314/234/base 2025-10-10T01:23:14.8074323Z * [new branch] gh/shunting314/234/head -> origin/gh/shunting314/234/head 2025-10-10T01:23:14.8076058Z * [new branch] gh/shunting314/234/orig -> origin/gh/shunting314/234/orig 2025-10-10T01:23:14.8078475Z * [new branch] gh/shunting314/235/base -> origin/gh/shunting314/235/base 2025-10-10T01:23:14.8080319Z * [new branch] gh/shunting314/235/head -> origin/gh/shunting314/235/head 2025-10-10T01:23:14.8082091Z * [new branch] gh/shunting314/235/orig -> origin/gh/shunting314/235/orig 2025-10-10T01:23:14.8084810Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-10-10T01:23:14.8086454Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-10-10T01:23:14.8088745Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-10-10T01:23:14.8090349Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-10-10T01:23:14.8092574Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-10-10T01:23:14.8094237Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-10-10T01:23:14.8096853Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-10-10T01:23:14.8099122Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-10-10T01:23:14.8101851Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-10-10T01:23:14.8103565Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-10-10T01:23:14.8106278Z * [new branch] gh/slayton58/1/base -> origin/gh/slayton58/1/base 2025-10-10T01:23:14.8108030Z * [new branch] gh/slayton58/1/head -> origin/gh/slayton58/1/head 2025-10-10T01:23:14.8109709Z * [new branch] gh/slayton58/1/orig -> origin/gh/slayton58/1/orig 2025-10-10T01:23:14.8111982Z * [new branch] gh/slayton58/10/base -> origin/gh/slayton58/10/base 2025-10-10T01:23:14.8113668Z * [new branch] gh/slayton58/10/head -> origin/gh/slayton58/10/head 2025-10-10T01:23:14.8115416Z * [new branch] gh/slayton58/10/orig -> origin/gh/slayton58/10/orig 2025-10-10T01:23:14.8117702Z * [new branch] gh/slayton58/11/base -> origin/gh/slayton58/11/base 2025-10-10T01:23:14.8119609Z * [new branch] gh/slayton58/11/head -> origin/gh/slayton58/11/head 2025-10-10T01:23:14.8121396Z * [new branch] gh/slayton58/11/orig -> origin/gh/slayton58/11/orig 2025-10-10T01:23:14.8123719Z * [new branch] gh/slayton58/12/base -> origin/gh/slayton58/12/base 2025-10-10T01:23:14.8125400Z * [new branch] gh/slayton58/12/head -> origin/gh/slayton58/12/head 2025-10-10T01:23:14.8127110Z * [new branch] gh/slayton58/12/orig -> origin/gh/slayton58/12/orig 2025-10-10T01:23:14.8129372Z * [new branch] gh/slayton58/13/base -> origin/gh/slayton58/13/base 2025-10-10T01:23:14.8131174Z * [new branch] gh/slayton58/13/head -> origin/gh/slayton58/13/head 2025-10-10T01:23:14.8132890Z * [new branch] gh/slayton58/13/orig -> origin/gh/slayton58/13/orig 2025-10-10T01:23:14.8135155Z * [new branch] gh/slayton58/14/base -> origin/gh/slayton58/14/base 2025-10-10T01:23:14.8136846Z * [new branch] gh/slayton58/14/head -> origin/gh/slayton58/14/head 2025-10-10T01:23:14.8138527Z * [new branch] gh/slayton58/14/orig -> origin/gh/slayton58/14/orig 2025-10-10T01:23:14.8140876Z * [new branch] gh/slayton58/15/base -> origin/gh/slayton58/15/base 2025-10-10T01:23:14.8142505Z * [new branch] gh/slayton58/15/head -> origin/gh/slayton58/15/head 2025-10-10T01:23:14.8144238Z * [new branch] gh/slayton58/15/orig -> origin/gh/slayton58/15/orig 2025-10-10T01:23:14.8146611Z * [new branch] gh/slayton58/16/base -> origin/gh/slayton58/16/base 2025-10-10T01:23:14.8148297Z * [new branch] gh/slayton58/16/head -> origin/gh/slayton58/16/head 2025-10-10T01:23:14.8149999Z * [new branch] gh/slayton58/16/orig -> origin/gh/slayton58/16/orig 2025-10-10T01:23:14.8152328Z * [new branch] gh/slayton58/17/base -> origin/gh/slayton58/17/base 2025-10-10T01:23:14.8154068Z * [new branch] gh/slayton58/17/head -> origin/gh/slayton58/17/head 2025-10-10T01:23:14.8155806Z * [new branch] gh/slayton58/17/orig -> origin/gh/slayton58/17/orig 2025-10-10T01:23:14.8158067Z * [new branch] gh/slayton58/18/base -> origin/gh/slayton58/18/base 2025-10-10T01:23:14.8159900Z * [new branch] gh/slayton58/18/head -> origin/gh/slayton58/18/head 2025-10-10T01:23:14.8162256Z * [new branch] gh/slayton58/19/base -> origin/gh/slayton58/19/base 2025-10-10T01:23:14.8163892Z * [new branch] gh/slayton58/19/head -> origin/gh/slayton58/19/head 2025-10-10T01:23:14.8165697Z * [new branch] gh/slayton58/19/orig -> origin/gh/slayton58/19/orig 2025-10-10T01:23:14.8167838Z * [new branch] gh/slayton58/2/base -> origin/gh/slayton58/2/base 2025-10-10T01:23:14.8169530Z * [new branch] gh/slayton58/2/head -> origin/gh/slayton58/2/head 2025-10-10T01:23:14.8171327Z * [new branch] gh/slayton58/2/orig -> origin/gh/slayton58/2/orig 2025-10-10T01:23:14.8173667Z * [new branch] gh/slayton58/20/base -> origin/gh/slayton58/20/base 2025-10-10T01:23:14.8175379Z * [new branch] gh/slayton58/20/head -> origin/gh/slayton58/20/head 2025-10-10T01:23:14.8177216Z * [new branch] gh/slayton58/20/orig -> origin/gh/slayton58/20/orig 2025-10-10T01:23:14.8179473Z * [new branch] gh/slayton58/21/base -> origin/gh/slayton58/21/base 2025-10-10T01:23:14.8181108Z * [new branch] gh/slayton58/21/head -> origin/gh/slayton58/21/head 2025-10-10T01:23:14.8182728Z * [new branch] gh/slayton58/21/orig -> origin/gh/slayton58/21/orig 2025-10-10T01:23:14.8185119Z * [new branch] gh/slayton58/22/base -> origin/gh/slayton58/22/base 2025-10-10T01:23:14.8186751Z * [new branch] gh/slayton58/22/head -> origin/gh/slayton58/22/head 2025-10-10T01:23:14.8188463Z * [new branch] gh/slayton58/22/orig -> origin/gh/slayton58/22/orig 2025-10-10T01:23:14.8190681Z * [new branch] gh/slayton58/23/base -> origin/gh/slayton58/23/base 2025-10-10T01:23:14.8192377Z * [new branch] gh/slayton58/23/head -> origin/gh/slayton58/23/head 2025-10-10T01:23:14.8194069Z * [new branch] gh/slayton58/23/orig -> origin/gh/slayton58/23/orig 2025-10-10T01:23:14.8196493Z * [new branch] gh/slayton58/24/base -> origin/gh/slayton58/24/base 2025-10-10T01:23:14.8198693Z * [new branch] gh/slayton58/24/head -> origin/gh/slayton58/24/head 2025-10-10T01:23:14.8200575Z * [new branch] gh/slayton58/24/orig -> origin/gh/slayton58/24/orig 2025-10-10T01:23:14.8202737Z * [new branch] gh/slayton58/25/base -> origin/gh/slayton58/25/base 2025-10-10T01:23:14.8204404Z * [new branch] gh/slayton58/25/head -> origin/gh/slayton58/25/head 2025-10-10T01:23:14.8206190Z * [new branch] gh/slayton58/25/orig -> origin/gh/slayton58/25/orig 2025-10-10T01:23:14.8209136Z * [new branch] gh/slayton58/26/base -> origin/gh/slayton58/26/base 2025-10-10T01:23:14.8211245Z * [new branch] gh/slayton58/26/head -> origin/gh/slayton58/26/head 2025-10-10T01:23:14.8213008Z * [new branch] gh/slayton58/26/orig -> origin/gh/slayton58/26/orig 2025-10-10T01:23:14.8215285Z * [new branch] gh/slayton58/3/base -> origin/gh/slayton58/3/base 2025-10-10T01:23:14.8216988Z * [new branch] gh/slayton58/3/head -> origin/gh/slayton58/3/head 2025-10-10T01:23:14.8218689Z * [new branch] gh/slayton58/3/orig -> origin/gh/slayton58/3/orig 2025-10-10T01:23:14.8220935Z * [new branch] gh/slayton58/4/base -> origin/gh/slayton58/4/base 2025-10-10T01:23:14.8222616Z * [new branch] gh/slayton58/4/head -> origin/gh/slayton58/4/head 2025-10-10T01:23:14.8224386Z * [new branch] gh/slayton58/4/orig -> origin/gh/slayton58/4/orig 2025-10-10T01:23:14.8226722Z * [new branch] gh/slayton58/5/base -> origin/gh/slayton58/5/base 2025-10-10T01:23:14.8228469Z * [new branch] gh/slayton58/5/head -> origin/gh/slayton58/5/head 2025-10-10T01:23:14.8230173Z * [new branch] gh/slayton58/5/orig -> origin/gh/slayton58/5/orig 2025-10-10T01:23:14.8232508Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-10-10T01:23:14.8234382Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-10-10T01:23:14.8236567Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-10-10T01:23:14.8238185Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-10-10T01:23:14.8241035Z * [new branch] gh/slayton58/8/base -> origin/gh/slayton58/8/base 2025-10-10T01:23:14.8242726Z * [new branch] gh/slayton58/8/head -> origin/gh/slayton58/8/head 2025-10-10T01:23:14.8244453Z * [new branch] gh/slayton58/8/orig -> origin/gh/slayton58/8/orig 2025-10-10T01:23:14.8246781Z * [new branch] gh/slayton58/9/base -> origin/gh/slayton58/9/base 2025-10-10T01:23:14.8248348Z * [new branch] gh/slayton58/9/head -> origin/gh/slayton58/9/head 2025-10-10T01:23:14.8250075Z * [new branch] gh/slayton58/9/orig -> origin/gh/slayton58/9/orig 2025-10-10T01:23:14.8253070Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-10-10T01:23:14.8254717Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-10-10T01:23:14.8256519Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-10-10T01:23:14.8259255Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-10-10T01:23:14.8260951Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-10-10T01:23:14.8262613Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-10-10T01:23:14.8265227Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-10-10T01:23:14.8266898Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-10-10T01:23:14.8268610Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-10-10T01:23:14.8271058Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-10-10T01:23:14.8272824Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-10-10T01:23:14.8274536Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-10-10T01:23:14.8276846Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-10-10T01:23:14.8278603Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-10-10T01:23:14.8280816Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-10-10T01:23:14.8283063Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-10-10T01:23:14.8284991Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-10-10T01:23:14.8286718Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-10-10T01:23:14.8289178Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-10-10T01:23:14.8291000Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-10-10T01:23:14.8292600Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-10-10T01:23:14.8294979Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-10-10T01:23:14.8296831Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-10-10T01:23:14.8298610Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-10-10T01:23:14.8300872Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-10-10T01:23:14.8302619Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-10-10T01:23:14.8304410Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-10-10T01:23:14.8306701Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-10-10T01:23:14.8308326Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-10-10T01:23:14.8309998Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-10-10T01:23:14.8312307Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-10-10T01:23:14.8314410Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-10-10T01:23:14.8316054Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-10-10T01:23:14.8318610Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-10-10T01:23:14.8320409Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-10-10T01:23:14.8322125Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-10-10T01:23:14.8324613Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-10-10T01:23:14.8326303Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-10-10T01:23:14.8327961Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-10-10T01:23:14.8330222Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-10-10T01:23:14.8331923Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-10-10T01:23:14.8333732Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-10-10T01:23:14.8336452Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-10-10T01:23:14.8338154Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-10-10T01:23:14.8339819Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-10-10T01:23:14.8342184Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-10-10T01:23:14.8344023Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-10-10T01:23:14.8345711Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-10-10T01:23:14.8348598Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-10-10T01:23:14.8350367Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-10-10T01:23:14.8352130Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-10-10T01:23:14.8354876Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-10-10T01:23:14.8356605Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-10-10T01:23:14.8358364Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-10-10T01:23:14.8361370Z * [new branch] gh/soulitzer/372/base -> origin/gh/soulitzer/372/base 2025-10-10T01:23:14.8363032Z * [new branch] gh/soulitzer/372/head -> origin/gh/soulitzer/372/head 2025-10-10T01:23:14.8364772Z * [new branch] gh/soulitzer/372/orig -> origin/gh/soulitzer/372/orig 2025-10-10T01:23:14.8367187Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-10-10T01:23:14.8368877Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-10-10T01:23:14.8370526Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-10-10T01:23:14.8373459Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-10-10T01:23:14.8375032Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-10-10T01:23:14.8376652Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-10-10T01:23:14.8378980Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-10-10T01:23:14.8380748Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-10-10T01:23:14.8382403Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-10-10T01:23:14.8384734Z * [new branch] gh/soulitzer/381/base -> origin/gh/soulitzer/381/base 2025-10-10T01:23:14.8386419Z * [new branch] gh/soulitzer/381/head -> origin/gh/soulitzer/381/head 2025-10-10T01:23:14.8388181Z * [new branch] gh/soulitzer/381/orig -> origin/gh/soulitzer/381/orig 2025-10-10T01:23:14.8390580Z * [new branch] gh/soulitzer/382/base -> origin/gh/soulitzer/382/base 2025-10-10T01:23:14.8392331Z * [new branch] gh/soulitzer/382/head -> origin/gh/soulitzer/382/head 2025-10-10T01:23:14.8394034Z * [new branch] gh/soulitzer/382/orig -> origin/gh/soulitzer/382/orig 2025-10-10T01:23:14.8396562Z * [new branch] gh/soulitzer/383/base -> origin/gh/soulitzer/383/base 2025-10-10T01:23:14.8398315Z * [new branch] gh/soulitzer/383/head -> origin/gh/soulitzer/383/head 2025-10-10T01:23:14.8400123Z * [new branch] gh/soulitzer/383/orig -> origin/gh/soulitzer/383/orig 2025-10-10T01:23:14.8402435Z * [new branch] gh/soulitzer/384/base -> origin/gh/soulitzer/384/base 2025-10-10T01:23:14.8404163Z * [new branch] gh/soulitzer/384/head -> origin/gh/soulitzer/384/head 2025-10-10T01:23:14.8405854Z * [new branch] gh/soulitzer/384/orig -> origin/gh/soulitzer/384/orig 2025-10-10T01:23:14.8408716Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-10-10T01:23:14.8411070Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-10-10T01:23:14.8412728Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-10-10T01:23:14.8414395Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-10-10T01:23:14.8416632Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-10-10T01:23:14.8418285Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-10-10T01:23:14.8419975Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-10-10T01:23:14.8422339Z * [new branch] gh/swolchok/809/base -> origin/gh/swolchok/809/base 2025-10-10T01:23:14.8424079Z * [new branch] gh/swolchok/809/head -> origin/gh/swolchok/809/head 2025-10-10T01:23:14.8425758Z * [new branch] gh/swolchok/809/orig -> origin/gh/swolchok/809/orig 2025-10-10T01:23:14.8428181Z * [new branch] gh/swolchok/815/base -> origin/gh/swolchok/815/base 2025-10-10T01:23:14.8429809Z * [new branch] gh/swolchok/815/head -> origin/gh/swolchok/815/head 2025-10-10T01:23:14.8431539Z * [new branch] gh/swolchok/815/orig -> origin/gh/swolchok/815/orig 2025-10-10T01:23:14.8434247Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-10-10T01:23:14.8435880Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-10-10T01:23:14.8437528Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-10-10T01:23:14.8440087Z * [new branch] gh/swolchok/821/base -> origin/gh/swolchok/821/base 2025-10-10T01:23:14.8441949Z * [new branch] gh/swolchok/821/head -> origin/gh/swolchok/821/head 2025-10-10T01:23:14.8443570Z * [new branch] gh/swolchok/821/orig -> origin/gh/swolchok/821/orig 2025-10-10T01:23:14.8445966Z * [new branch] gh/swolchok/823/base -> origin/gh/swolchok/823/base 2025-10-10T01:23:14.8447601Z * [new branch] gh/swolchok/823/head -> origin/gh/swolchok/823/head 2025-10-10T01:23:14.8449341Z * [new branch] gh/swolchok/823/orig -> origin/gh/swolchok/823/orig 2025-10-10T01:23:14.8451567Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-10-10T01:23:14.8453693Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-10-10T01:23:14.8455406Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-10-10T01:23:14.8457802Z * [new branch] gh/swolchok/826/base -> origin/gh/swolchok/826/base 2025-10-10T01:23:14.8459562Z * [new branch] gh/swolchok/826/head -> origin/gh/swolchok/826/head 2025-10-10T01:23:14.8461324Z * [new branch] gh/swolchok/826/orig -> origin/gh/swolchok/826/orig 2025-10-10T01:23:14.8463708Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-10-10T01:23:14.8465356Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-10-10T01:23:14.8467076Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-10-10T01:23:14.8469467Z * [new branch] gh/swolchok/830/base -> origin/gh/swolchok/830/base 2025-10-10T01:23:14.8471133Z * [new branch] gh/swolchok/830/head -> origin/gh/swolchok/830/head 2025-10-10T01:23:14.8472965Z * [new branch] gh/swolchok/830/orig -> origin/gh/swolchok/830/orig 2025-10-10T01:23:14.8475293Z * [new branch] gh/swolchok/831/base -> origin/gh/swolchok/831/base 2025-10-10T01:23:14.8476918Z * [new branch] gh/swolchok/831/head -> origin/gh/swolchok/831/head 2025-10-10T01:23:14.8478613Z * [new branch] gh/swolchok/831/orig -> origin/gh/swolchok/831/orig 2025-10-10T01:23:14.8481180Z * [new branch] gh/swolchok/832/base -> origin/gh/swolchok/832/base 2025-10-10T01:23:14.8482796Z * [new branch] gh/swolchok/832/head -> origin/gh/swolchok/832/head 2025-10-10T01:23:14.8484602Z * [new branch] gh/swolchok/832/orig -> origin/gh/swolchok/832/orig 2025-10-10T01:23:14.8486922Z * [new branch] gh/swolchok/833/base -> origin/gh/swolchok/833/base 2025-10-10T01:23:14.8488767Z * [new branch] gh/swolchok/833/head -> origin/gh/swolchok/833/head 2025-10-10T01:23:14.8490465Z * [new branch] gh/swolchok/833/orig -> origin/gh/swolchok/833/orig 2025-10-10T01:23:14.8492835Z * [new branch] gh/swolchok/834/base -> origin/gh/swolchok/834/base 2025-10-10T01:23:14.8494472Z * [new branch] gh/swolchok/834/head -> origin/gh/swolchok/834/head 2025-10-10T01:23:14.8496288Z * [new branch] gh/swolchok/834/orig -> origin/gh/swolchok/834/orig 2025-10-10T01:23:14.8500529Z * [new branch] gh/swolchok/835/base -> origin/gh/swolchok/835/base 2025-10-10T01:23:14.8502249Z * [new branch] gh/swolchok/835/head -> origin/gh/swolchok/835/head 2025-10-10T01:23:14.8504064Z * [new branch] gh/swolchok/835/orig -> origin/gh/swolchok/835/orig 2025-10-10T01:23:14.8506640Z * [new branch] gh/swolchok/836/base -> origin/gh/swolchok/836/base 2025-10-10T01:23:14.8508403Z * [new branch] gh/swolchok/836/head -> origin/gh/swolchok/836/head 2025-10-10T01:23:14.8510108Z * [new branch] gh/swolchok/836/orig -> origin/gh/swolchok/836/orig 2025-10-10T01:23:14.8512593Z * [new branch] gh/swolchok/837/base -> origin/gh/swolchok/837/base 2025-10-10T01:23:14.8514153Z * [new branch] gh/swolchok/837/head -> origin/gh/swolchok/837/head 2025-10-10T01:23:14.8515788Z * [new branch] gh/swolchok/837/orig -> origin/gh/swolchok/837/orig 2025-10-10T01:23:14.8518235Z * [new branch] gh/swolchok/838/base -> origin/gh/swolchok/838/base 2025-10-10T01:23:14.8520039Z * [new branch] gh/swolchok/838/head -> origin/gh/swolchok/838/head 2025-10-10T01:23:14.8521804Z * [new branch] gh/swolchok/838/orig -> origin/gh/swolchok/838/orig 2025-10-10T01:23:14.8524215Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-10-10T01:23:14.8525839Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-10-10T01:23:14.8527526Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-10-10T01:23:14.8530065Z * [new branch] gh/swolchok/840/base -> origin/gh/swolchok/840/base 2025-10-10T01:23:14.8531769Z * [new branch] gh/swolchok/840/head -> origin/gh/swolchok/840/head 2025-10-10T01:23:14.8533587Z * [new branch] gh/swolchok/840/orig -> origin/gh/swolchok/840/orig 2025-10-10T01:23:14.8535909Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-10-10T01:23:14.8537593Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-10-10T01:23:14.8539287Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-10-10T01:23:14.8541656Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-10-10T01:23:14.8543414Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-10-10T01:23:14.8545101Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-10-10T01:23:14.8547559Z * [new branch] gh/swolchok/843/base -> origin/gh/swolchok/843/base 2025-10-10T01:23:14.8549245Z * [new branch] gh/swolchok/843/head -> origin/gh/swolchok/843/head 2025-10-10T01:23:14.8550980Z * [new branch] gh/swolchok/843/orig -> origin/gh/swolchok/843/orig 2025-10-10T01:23:14.8553430Z * [new branch] gh/swolchok/844/base -> origin/gh/swolchok/844/base 2025-10-10T01:23:14.8555065Z * [new branch] gh/swolchok/844/head -> origin/gh/swolchok/844/head 2025-10-10T01:23:14.8556837Z * [new branch] gh/swolchok/844/orig -> origin/gh/swolchok/844/orig 2025-10-10T01:23:14.8559365Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-10-10T01:23:14.8561108Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-10-10T01:23:14.8562841Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-10-10T01:23:14.8565272Z * [new branch] gh/swolchok/846/base -> origin/gh/swolchok/846/base 2025-10-10T01:23:14.8567123Z * [new branch] gh/swolchok/846/head -> origin/gh/swolchok/846/head 2025-10-10T01:23:14.8568820Z * [new branch] gh/swolchok/846/orig -> origin/gh/swolchok/846/orig 2025-10-10T01:23:14.8571326Z * [new branch] gh/swolchok/847/base -> origin/gh/swolchok/847/base 2025-10-10T01:23:14.8572959Z * [new branch] gh/swolchok/847/head -> origin/gh/swolchok/847/head 2025-10-10T01:23:14.8574654Z * [new branch] gh/swolchok/847/orig -> origin/gh/swolchok/847/orig 2025-10-10T01:23:14.8577084Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-10-10T01:23:14.8578839Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-10-10T01:23:14.8580642Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-10-10T01:23:14.8583102Z * [new branch] gh/swolchok/849/base -> origin/gh/swolchok/849/base 2025-10-10T01:23:14.8584779Z * [new branch] gh/swolchok/849/head -> origin/gh/swolchok/849/head 2025-10-10T01:23:14.8586551Z * [new branch] gh/swolchok/849/orig -> origin/gh/swolchok/849/orig 2025-10-10T01:23:14.8588774Z * [new branch] gh/swolchok/850/base -> origin/gh/swolchok/850/base 2025-10-10T01:23:14.8590545Z * [new branch] gh/swolchok/850/head -> origin/gh/swolchok/850/head 2025-10-10T01:23:14.8592273Z * [new branch] gh/swolchok/850/orig -> origin/gh/swolchok/850/orig 2025-10-10T01:23:14.8594673Z * [new branch] gh/swolchok/851/base -> origin/gh/swolchok/851/base 2025-10-10T01:23:14.8596778Z * [new branch] gh/swolchok/851/head -> origin/gh/swolchok/851/head 2025-10-10T01:23:14.8598795Z * [new branch] gh/swolchok/851/orig -> origin/gh/swolchok/851/orig 2025-10-10T01:23:14.8601457Z * [new branch] gh/swolchok/852/base -> origin/gh/swolchok/852/base 2025-10-10T01:23:14.8603113Z * [new branch] gh/swolchok/852/head -> origin/gh/swolchok/852/head 2025-10-10T01:23:14.8605229Z * [new branch] gh/swolchok/852/orig -> origin/gh/swolchok/852/orig 2025-10-10T01:23:14.8608041Z * [new branch] gh/syed-ahmed/5/base -> origin/gh/syed-ahmed/5/base 2025-10-10T01:23:14.8609712Z * [new branch] gh/syed-ahmed/5/head -> origin/gh/syed-ahmed/5/head 2025-10-10T01:23:14.8611559Z * [new branch] gh/syed-ahmed/5/orig -> origin/gh/syed-ahmed/5/orig 2025-10-10T01:23:14.8613839Z * [new branch] gh/syed-ahmed/6/base -> origin/gh/syed-ahmed/6/base 2025-10-10T01:23:14.8615608Z * [new branch] gh/syed-ahmed/6/head -> origin/gh/syed-ahmed/6/head 2025-10-10T01:23:14.8617439Z * [new branch] gh/syed-ahmed/6/orig -> origin/gh/syed-ahmed/6/orig 2025-10-10T01:23:14.8619653Z * [new branch] gh/syed-ahmed/7/base -> origin/gh/syed-ahmed/7/base 2025-10-10T01:23:14.8621255Z * [new branch] gh/syed-ahmed/7/head -> origin/gh/syed-ahmed/7/head 2025-10-10T01:23:14.8622950Z * [new branch] gh/syed-ahmed/7/orig -> origin/gh/syed-ahmed/7/orig 2025-10-10T01:23:14.8625826Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-10-10T01:23:14.8627598Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-10-10T01:23:14.8629307Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-10-10T01:23:14.8632477Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-10-10T01:23:14.8634323Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-10-10T01:23:14.8636008Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-10-10T01:23:14.8638283Z * [new branch] gh/tianyu-l/5/base -> origin/gh/tianyu-l/5/base 2025-10-10T01:23:14.8640599Z * [new branch] gh/tianyu-l/5/orig -> origin/gh/tianyu-l/5/orig 2025-10-10T01:23:14.8642883Z * [new branch] gh/tianyu-l/6/base -> origin/gh/tianyu-l/6/base 2025-10-10T01:23:14.8644583Z * [new branch] gh/tianyu-l/6/head -> origin/gh/tianyu-l/6/head 2025-10-10T01:23:14.8646368Z * [new branch] gh/tianyu-l/6/orig -> origin/gh/tianyu-l/6/orig 2025-10-10T01:23:14.8648684Z * [new branch] gh/tianyu-l/7/base -> origin/gh/tianyu-l/7/base 2025-10-10T01:23:14.8650370Z * [new branch] gh/tianyu-l/7/orig -> origin/gh/tianyu-l/7/orig 2025-10-10T01:23:14.8653747Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-10-10T01:23:14.8655317Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-10-10T01:23:14.8656979Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-10-10T01:23:14.8659340Z * [new branch] gh/tugsbayasgalan/11/base -> origin/gh/tugsbayasgalan/11/base 2025-10-10T01:23:14.8661052Z * [new branch] gh/tugsbayasgalan/11/head -> origin/gh/tugsbayasgalan/11/head 2025-10-10T01:23:14.8662742Z * [new branch] gh/tugsbayasgalan/11/orig -> origin/gh/tugsbayasgalan/11/orig 2025-10-10T01:23:14.8665071Z * [new branch] gh/tugsbayasgalan/12/base -> origin/gh/tugsbayasgalan/12/base 2025-10-10T01:23:14.8666731Z * [new branch] gh/tugsbayasgalan/12/head -> origin/gh/tugsbayasgalan/12/head 2025-10-10T01:23:14.8668535Z * [new branch] gh/tugsbayasgalan/12/orig -> origin/gh/tugsbayasgalan/12/orig 2025-10-10T01:23:14.8670900Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-10-10T01:23:14.8672498Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-10-10T01:23:14.8674284Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-10-10T01:23:14.8676600Z * [new branch] gh/tugsbayasgalan/14/base -> origin/gh/tugsbayasgalan/14/base 2025-10-10T01:23:14.8678405Z * [new branch] gh/tugsbayasgalan/14/head -> origin/gh/tugsbayasgalan/14/head 2025-10-10T01:23:14.8680255Z * [new branch] gh/tugsbayasgalan/14/orig -> origin/gh/tugsbayasgalan/14/orig 2025-10-10T01:23:14.8682824Z * [new branch] gh/tugsbayasgalan/15/base -> origin/gh/tugsbayasgalan/15/base 2025-10-10T01:23:14.8684391Z * [new branch] gh/tugsbayasgalan/15/head -> origin/gh/tugsbayasgalan/15/head 2025-10-10T01:23:14.8686137Z * [new branch] gh/tugsbayasgalan/15/orig -> origin/gh/tugsbayasgalan/15/orig 2025-10-10T01:23:14.8688448Z * [new branch] gh/tugsbayasgalan/16/base -> origin/gh/tugsbayasgalan/16/base 2025-10-10T01:23:14.8690167Z * [new branch] gh/tugsbayasgalan/16/head -> origin/gh/tugsbayasgalan/16/head 2025-10-10T01:23:14.8691933Z * [new branch] gh/tugsbayasgalan/16/orig -> origin/gh/tugsbayasgalan/16/orig 2025-10-10T01:23:14.8694436Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-10-10T01:23:14.8696219Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-10-10T01:23:14.8698028Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-10-10T01:23:14.8700353Z * [new branch] gh/tugsbayasgalan/18/base -> origin/gh/tugsbayasgalan/18/base 2025-10-10T01:23:14.8702024Z * [new branch] gh/tugsbayasgalan/18/head -> origin/gh/tugsbayasgalan/18/head 2025-10-10T01:23:14.8703771Z * [new branch] gh/tugsbayasgalan/18/orig -> origin/gh/tugsbayasgalan/18/orig 2025-10-10T01:23:14.8706052Z * [new branch] gh/tugsbayasgalan/19/base -> origin/gh/tugsbayasgalan/19/base 2025-10-10T01:23:14.8707787Z * [new branch] gh/tugsbayasgalan/19/head -> origin/gh/tugsbayasgalan/19/head 2025-10-10T01:23:14.8709469Z * [new branch] gh/tugsbayasgalan/19/orig -> origin/gh/tugsbayasgalan/19/orig 2025-10-10T01:23:14.8711706Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-10-10T01:23:14.8713411Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-10-10T01:23:14.8715182Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-10-10T01:23:14.8717562Z * [new branch] gh/tugsbayasgalan/20/base -> origin/gh/tugsbayasgalan/20/base 2025-10-10T01:23:14.8719542Z * [new branch] gh/tugsbayasgalan/20/head -> origin/gh/tugsbayasgalan/20/head 2025-10-10T01:23:14.8721129Z * [new branch] gh/tugsbayasgalan/20/orig -> origin/gh/tugsbayasgalan/20/orig 2025-10-10T01:23:14.8723781Z * [new branch] gh/tugsbayasgalan/21/base -> origin/gh/tugsbayasgalan/21/base 2025-10-10T01:23:14.8725555Z * [new branch] gh/tugsbayasgalan/21/head -> origin/gh/tugsbayasgalan/21/head 2025-10-10T01:23:14.8727337Z * [new branch] gh/tugsbayasgalan/21/orig -> origin/gh/tugsbayasgalan/21/orig 2025-10-10T01:23:14.8729645Z * [new branch] gh/tugsbayasgalan/22/base -> origin/gh/tugsbayasgalan/22/base 2025-10-10T01:23:14.8731476Z * [new branch] gh/tugsbayasgalan/22/head -> origin/gh/tugsbayasgalan/22/head 2025-10-10T01:23:14.8733234Z * [new branch] gh/tugsbayasgalan/22/orig -> origin/gh/tugsbayasgalan/22/orig 2025-10-10T01:23:14.8735949Z * [new branch] gh/tugsbayasgalan/23/base -> origin/gh/tugsbayasgalan/23/base 2025-10-10T01:23:14.8737653Z * [new branch] gh/tugsbayasgalan/23/head -> origin/gh/tugsbayasgalan/23/head 2025-10-10T01:23:14.8739425Z * [new branch] gh/tugsbayasgalan/23/orig -> origin/gh/tugsbayasgalan/23/orig 2025-10-10T01:23:14.8741896Z * [new branch] gh/tugsbayasgalan/24/base -> origin/gh/tugsbayasgalan/24/base 2025-10-10T01:23:14.8743575Z * [new branch] gh/tugsbayasgalan/24/head -> origin/gh/tugsbayasgalan/24/head 2025-10-10T01:23:14.8745241Z * [new branch] gh/tugsbayasgalan/24/orig -> origin/gh/tugsbayasgalan/24/orig 2025-10-10T01:23:14.8747835Z * [new branch] gh/tugsbayasgalan/25/base -> origin/gh/tugsbayasgalan/25/base 2025-10-10T01:23:14.8749508Z * [new branch] gh/tugsbayasgalan/25/head -> origin/gh/tugsbayasgalan/25/head 2025-10-10T01:23:14.8751169Z * [new branch] gh/tugsbayasgalan/25/orig -> origin/gh/tugsbayasgalan/25/orig 2025-10-10T01:23:14.8753597Z * [new branch] gh/tugsbayasgalan/26/base -> origin/gh/tugsbayasgalan/26/base 2025-10-10T01:23:14.8755411Z * [new branch] gh/tugsbayasgalan/26/head -> origin/gh/tugsbayasgalan/26/head 2025-10-10T01:23:14.8757089Z * [new branch] gh/tugsbayasgalan/26/orig -> origin/gh/tugsbayasgalan/26/orig 2025-10-10T01:23:14.8760350Z * [new branch] gh/tugsbayasgalan/27/base -> origin/gh/tugsbayasgalan/27/base 2025-10-10T01:23:14.8762100Z * [new branch] gh/tugsbayasgalan/27/head -> origin/gh/tugsbayasgalan/27/head 2025-10-10T01:23:14.8763933Z * [new branch] gh/tugsbayasgalan/27/orig -> origin/gh/tugsbayasgalan/27/orig 2025-10-10T01:23:14.8766500Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-10-10T01:23:14.8768225Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-10-10T01:23:14.8769970Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-10-10T01:23:14.8772446Z * [new branch] gh/tugsbayasgalan/29/base -> origin/gh/tugsbayasgalan/29/base 2025-10-10T01:23:14.8774208Z * [new branch] gh/tugsbayasgalan/29/head -> origin/gh/tugsbayasgalan/29/head 2025-10-10T01:23:14.8776312Z * [new branch] gh/tugsbayasgalan/29/orig -> origin/gh/tugsbayasgalan/29/orig 2025-10-10T01:23:14.8778677Z * [new branch] gh/tugsbayasgalan/3/base -> origin/gh/tugsbayasgalan/3/base 2025-10-10T01:23:14.8780552Z * [new branch] gh/tugsbayasgalan/3/head -> origin/gh/tugsbayasgalan/3/head 2025-10-10T01:23:14.8782213Z * [new branch] gh/tugsbayasgalan/3/orig -> origin/gh/tugsbayasgalan/3/orig 2025-10-10T01:23:14.8784592Z * [new branch] gh/tugsbayasgalan/30/base -> origin/gh/tugsbayasgalan/30/base 2025-10-10T01:23:14.8786330Z * [new branch] gh/tugsbayasgalan/30/head -> origin/gh/tugsbayasgalan/30/head 2025-10-10T01:23:14.8788726Z * [new branch] gh/tugsbayasgalan/30/orig -> origin/gh/tugsbayasgalan/30/orig 2025-10-10T01:23:14.8791046Z * [new branch] gh/tugsbayasgalan/31/base -> origin/gh/tugsbayasgalan/31/base 2025-10-10T01:23:14.8792820Z * [new branch] gh/tugsbayasgalan/31/head -> origin/gh/tugsbayasgalan/31/head 2025-10-10T01:23:14.8794575Z * [new branch] gh/tugsbayasgalan/31/orig -> origin/gh/tugsbayasgalan/31/orig 2025-10-10T01:23:14.8797094Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-10-10T01:23:14.8798787Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-10-10T01:23:14.8800664Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-10-10T01:23:14.8803044Z * [new branch] gh/tugsbayasgalan/33/base -> origin/gh/tugsbayasgalan/33/base 2025-10-10T01:23:14.8804863Z * [new branch] gh/tugsbayasgalan/33/head -> origin/gh/tugsbayasgalan/33/head 2025-10-10T01:23:14.8806487Z * [new branch] gh/tugsbayasgalan/33/orig -> origin/gh/tugsbayasgalan/33/orig 2025-10-10T01:23:14.8809022Z * [new branch] gh/tugsbayasgalan/34/base -> origin/gh/tugsbayasgalan/34/base 2025-10-10T01:23:14.8810695Z * [new branch] gh/tugsbayasgalan/34/head -> origin/gh/tugsbayasgalan/34/head 2025-10-10T01:23:14.8812499Z * [new branch] gh/tugsbayasgalan/34/orig -> origin/gh/tugsbayasgalan/34/orig 2025-10-10T01:23:14.8814753Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-10-10T01:23:14.8816558Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-10-10T01:23:14.8818205Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-10-10T01:23:14.8820601Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-10-10T01:23:14.8822283Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-10-10T01:23:14.8823980Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-10-10T01:23:14.8826400Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-10-10T01:23:14.8828115Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-10-10T01:23:14.8829768Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-10-10T01:23:14.8832408Z * [new branch] gh/tugsbayasgalan/38/base -> origin/gh/tugsbayasgalan/38/base 2025-10-10T01:23:14.8834226Z * [new branch] gh/tugsbayasgalan/38/head -> origin/gh/tugsbayasgalan/38/head 2025-10-10T01:23:14.8835970Z * [new branch] gh/tugsbayasgalan/38/orig -> origin/gh/tugsbayasgalan/38/orig 2025-10-10T01:23:14.8838337Z * [new branch] gh/tugsbayasgalan/39/base -> origin/gh/tugsbayasgalan/39/base 2025-10-10T01:23:14.8840151Z * [new branch] gh/tugsbayasgalan/39/head -> origin/gh/tugsbayasgalan/39/head 2025-10-10T01:23:14.8841943Z * [new branch] gh/tugsbayasgalan/39/orig -> origin/gh/tugsbayasgalan/39/orig 2025-10-10T01:23:14.8844404Z * [new branch] gh/tugsbayasgalan/40/base -> origin/gh/tugsbayasgalan/40/base 2025-10-10T01:23:14.8846013Z * [new branch] gh/tugsbayasgalan/40/head -> origin/gh/tugsbayasgalan/40/head 2025-10-10T01:23:14.8847674Z * [new branch] gh/tugsbayasgalan/40/orig -> origin/gh/tugsbayasgalan/40/orig 2025-10-10T01:23:14.8850192Z * [new branch] gh/tugsbayasgalan/41/base -> origin/gh/tugsbayasgalan/41/base 2025-10-10T01:23:14.8851906Z * [new branch] gh/tugsbayasgalan/41/head -> origin/gh/tugsbayasgalan/41/head 2025-10-10T01:23:14.8854007Z * [new branch] gh/tugsbayasgalan/41/orig -> origin/gh/tugsbayasgalan/41/orig 2025-10-10T01:23:14.8856404Z * [new branch] gh/tugsbayasgalan/42/base -> origin/gh/tugsbayasgalan/42/base 2025-10-10T01:23:14.8858327Z * [new branch] gh/tugsbayasgalan/42/head -> origin/gh/tugsbayasgalan/42/head 2025-10-10T01:23:14.8860037Z * [new branch] gh/tugsbayasgalan/42/orig -> origin/gh/tugsbayasgalan/42/orig 2025-10-10T01:23:14.8862824Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-10-10T01:23:14.8864566Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-10-10T01:23:14.8866314Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-10-10T01:23:14.8868696Z * [new branch] gh/tugsbayasgalan/44/base -> origin/gh/tugsbayasgalan/44/base 2025-10-10T01:23:14.8870371Z * [new branch] gh/tugsbayasgalan/44/head -> origin/gh/tugsbayasgalan/44/head 2025-10-10T01:23:14.8872041Z * [new branch] gh/tugsbayasgalan/44/orig -> origin/gh/tugsbayasgalan/44/orig 2025-10-10T01:23:14.8874440Z * [new branch] gh/tugsbayasgalan/45/base -> origin/gh/tugsbayasgalan/45/base 2025-10-10T01:23:14.8876267Z * [new branch] gh/tugsbayasgalan/45/head -> origin/gh/tugsbayasgalan/45/head 2025-10-10T01:23:14.8877936Z * [new branch] gh/tugsbayasgalan/45/orig -> origin/gh/tugsbayasgalan/45/orig 2025-10-10T01:23:14.8880630Z * [new branch] gh/tugsbayasgalan/46/base -> origin/gh/tugsbayasgalan/46/base 2025-10-10T01:23:14.8882264Z * [new branch] gh/tugsbayasgalan/46/head -> origin/gh/tugsbayasgalan/46/head 2025-10-10T01:23:14.8884035Z * [new branch] gh/tugsbayasgalan/46/orig -> origin/gh/tugsbayasgalan/46/orig 2025-10-10T01:23:14.8886490Z * [new branch] gh/tugsbayasgalan/47/base -> origin/gh/tugsbayasgalan/47/base 2025-10-10T01:23:14.8888271Z * [new branch] gh/tugsbayasgalan/47/head -> origin/gh/tugsbayasgalan/47/head 2025-10-10T01:23:14.8889963Z * [new branch] gh/tugsbayasgalan/47/orig -> origin/gh/tugsbayasgalan/47/orig 2025-10-10T01:23:14.8892202Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-10-10T01:23:14.8893870Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-10-10T01:23:14.8895557Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-10-10T01:23:14.8899296Z * [new branch] gh/tugsbayasgalan/49/base -> origin/gh/tugsbayasgalan/49/base 2025-10-10T01:23:14.8901222Z * [new branch] gh/tugsbayasgalan/49/head -> origin/gh/tugsbayasgalan/49/head 2025-10-10T01:23:14.8903056Z * [new branch] gh/tugsbayasgalan/49/orig -> origin/gh/tugsbayasgalan/49/orig 2025-10-10T01:23:14.8905399Z * [new branch] gh/tugsbayasgalan/50/base -> origin/gh/tugsbayasgalan/50/base 2025-10-10T01:23:14.8907159Z * [new branch] gh/tugsbayasgalan/50/head -> origin/gh/tugsbayasgalan/50/head 2025-10-10T01:23:14.8908999Z * [new branch] gh/tugsbayasgalan/50/orig -> origin/gh/tugsbayasgalan/50/orig 2025-10-10T01:23:14.8911148Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-10-10T01:23:14.8912861Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-10-10T01:23:14.8914601Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-10-10T01:23:14.8916841Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-10-10T01:23:14.8918540Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-10-10T01:23:14.8920410Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-10-10T01:23:14.8922920Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-10-10T01:23:14.8924473Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-10-10T01:23:14.8926128Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-10-10T01:23:14.8928388Z * [new branch] gh/tugsbayasgalan/54/base -> origin/gh/tugsbayasgalan/54/base 2025-10-10T01:23:14.8930150Z * [new branch] gh/tugsbayasgalan/54/head -> origin/gh/tugsbayasgalan/54/head 2025-10-10T01:23:14.8931946Z * [new branch] gh/tugsbayasgalan/54/orig -> origin/gh/tugsbayasgalan/54/orig 2025-10-10T01:23:14.8934244Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-10-10T01:23:14.8935914Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-10-10T01:23:14.8937597Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-10-10T01:23:14.8939987Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-10-10T01:23:14.8941797Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-10-10T01:23:14.8943619Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-10-10T01:23:14.8945939Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-10-10T01:23:14.8947582Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-10-10T01:23:14.8949234Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-10-10T01:23:14.8951577Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-10-10T01:23:14.8953229Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-10-10T01:23:14.8955005Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-10-10T01:23:14.8957724Z * [new branch] gh/v0i0/10/base -> origin/gh/v0i0/10/base 2025-10-10T01:23:14.8959606Z * [new branch] gh/v0i0/10/head -> origin/gh/v0i0/10/head 2025-10-10T01:23:14.8961199Z * [new branch] gh/v0i0/10/orig -> origin/gh/v0i0/10/orig 2025-10-10T01:23:14.8963460Z * [new branch] gh/v0i0/11/base -> origin/gh/v0i0/11/base 2025-10-10T01:23:14.8965575Z * [new branch] gh/v0i0/11/head -> origin/gh/v0i0/11/head 2025-10-10T01:23:14.8967300Z * [new branch] gh/v0i0/11/orig -> origin/gh/v0i0/11/orig 2025-10-10T01:23:14.8969777Z * [new branch] gh/v0i0/12/base -> origin/gh/v0i0/12/base 2025-10-10T01:23:14.8971566Z * [new branch] gh/v0i0/12/head -> origin/gh/v0i0/12/head 2025-10-10T01:23:14.8973404Z * [new branch] gh/v0i0/12/orig -> origin/gh/v0i0/12/orig 2025-10-10T01:23:14.8975838Z * [new branch] gh/v0i0/13/base -> origin/gh/v0i0/13/base 2025-10-10T01:23:14.8977532Z * [new branch] gh/v0i0/13/head -> origin/gh/v0i0/13/head 2025-10-10T01:23:14.8979211Z * [new branch] gh/v0i0/13/orig -> origin/gh/v0i0/13/orig 2025-10-10T01:23:14.8981539Z * [new branch] gh/v0i0/7/base -> origin/gh/v0i0/7/base 2025-10-10T01:23:14.8983311Z * [new branch] gh/v0i0/7/head -> origin/gh/v0i0/7/head 2025-10-10T01:23:14.8985025Z * [new branch] gh/v0i0/7/orig -> origin/gh/v0i0/7/orig 2025-10-10T01:23:14.8987287Z * [new branch] gh/v0i0/8/base -> origin/gh/v0i0/8/base 2025-10-10T01:23:14.8989009Z * [new branch] gh/v0i0/8/head -> origin/gh/v0i0/8/head 2025-10-10T01:23:14.8990813Z * [new branch] gh/v0i0/8/orig -> origin/gh/v0i0/8/orig 2025-10-10T01:23:14.8993066Z * [new branch] gh/v0i0/9/base -> origin/gh/v0i0/9/base 2025-10-10T01:23:14.8994789Z * [new branch] gh/v0i0/9/head -> origin/gh/v0i0/9/head 2025-10-10T01:23:14.8996650Z * [new branch] gh/v0i0/9/orig -> origin/gh/v0i0/9/orig 2025-10-10T01:23:14.8999817Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-10-10T01:23:14.9001692Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-10-10T01:23:14.9003864Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-10-10T01:23:14.9005579Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-10-10T01:23:14.9007267Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-10-10T01:23:14.9009998Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-10-10T01:23:14.9012228Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-10-10T01:23:14.9014572Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-10-10T01:23:14.9017807Z * [new branch] gh/vkuzo/7/base -> origin/gh/vkuzo/7/base 2025-10-10T01:23:14.9019510Z * [new branch] gh/vkuzo/7/head -> origin/gh/vkuzo/7/head 2025-10-10T01:23:14.9021274Z * [new branch] gh/vkuzo/7/orig -> origin/gh/vkuzo/7/orig 2025-10-10T01:23:14.9024278Z * [new branch] gh/wconstab/419/base -> origin/gh/wconstab/419/base 2025-10-10T01:23:14.9025855Z * [new branch] gh/wconstab/419/head -> origin/gh/wconstab/419/head 2025-10-10T01:23:14.9027664Z * [new branch] gh/wconstab/419/orig -> origin/gh/wconstab/419/orig 2025-10-10T01:23:14.9029931Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-10-10T01:23:14.9031656Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-10-10T01:23:14.9033336Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-10-10T01:23:14.9035678Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-10-10T01:23:14.9037412Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-10-10T01:23:14.9039172Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-10-10T01:23:14.9041604Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-10-10T01:23:14.9043311Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-10-10T01:23:14.9045003Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-10-10T01:23:14.9047241Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-10-10T01:23:14.9048990Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-10-10T01:23:14.9050692Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-10-10T01:23:14.9053127Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-10-10T01:23:14.9054929Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-10-10T01:23:14.9056646Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-10-10T01:23:14.9059436Z * [new branch] gh/weifengpy/30/base -> origin/gh/weifengpy/30/base 2025-10-10T01:23:14.9061112Z * [new branch] gh/weifengpy/30/head -> origin/gh/weifengpy/30/head 2025-10-10T01:23:14.9063067Z * [new branch] gh/weifengpy/30/orig -> origin/gh/weifengpy/30/orig 2025-10-10T01:23:14.9065285Z * [new branch] gh/weifengpy/31/base -> origin/gh/weifengpy/31/base 2025-10-10T01:23:14.9067100Z * [new branch] gh/weifengpy/31/head -> origin/gh/weifengpy/31/head 2025-10-10T01:23:14.9068790Z * [new branch] gh/weifengpy/31/orig -> origin/gh/weifengpy/31/orig 2025-10-10T01:23:14.9070930Z * [new branch] gh/weifengpy/32/base -> origin/gh/weifengpy/32/base 2025-10-10T01:23:14.9072882Z * [new branch] gh/weifengpy/32/head -> origin/gh/weifengpy/32/head 2025-10-10T01:23:14.9074561Z * [new branch] gh/weifengpy/32/orig -> origin/gh/weifengpy/32/orig 2025-10-10T01:23:14.9077200Z * [new branch] gh/weifengpy/33/base -> origin/gh/weifengpy/33/base 2025-10-10T01:23:14.9079057Z * [new branch] gh/weifengpy/33/head -> origin/gh/weifengpy/33/head 2025-10-10T01:23:14.9080899Z * [new branch] gh/weifengpy/33/orig -> origin/gh/weifengpy/33/orig 2025-10-10T01:23:14.9083302Z * [new branch] gh/weifengpy/34/base -> origin/gh/weifengpy/34/base 2025-10-10T01:23:14.9085007Z * [new branch] gh/weifengpy/34/head -> origin/gh/weifengpy/34/head 2025-10-10T01:23:14.9086724Z * [new branch] gh/weifengpy/34/orig -> origin/gh/weifengpy/34/orig 2025-10-10T01:23:14.9089117Z * [new branch] gh/weifengpy/35/base -> origin/gh/weifengpy/35/base 2025-10-10T01:23:14.9090735Z * [new branch] gh/weifengpy/35/head -> origin/gh/weifengpy/35/head 2025-10-10T01:23:14.9092575Z * [new branch] gh/weifengpy/35/orig -> origin/gh/weifengpy/35/orig 2025-10-10T01:23:14.9094958Z * [new branch] gh/weifengpy/36/base -> origin/gh/weifengpy/36/base 2025-10-10T01:23:14.9097201Z * [new branch] gh/weifengpy/36/head -> origin/gh/weifengpy/36/head 2025-10-10T01:23:14.9098942Z * [new branch] gh/weifengpy/36/orig -> origin/gh/weifengpy/36/orig 2025-10-10T01:23:14.9101044Z * [new branch] gh/weifengpy/37/base -> origin/gh/weifengpy/37/base 2025-10-10T01:23:14.9102739Z * [new branch] gh/weifengpy/37/head -> origin/gh/weifengpy/37/head 2025-10-10T01:23:14.9104410Z * [new branch] gh/weifengpy/37/orig -> origin/gh/weifengpy/37/orig 2025-10-10T01:23:14.9107905Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-10-10T01:23:14.9109589Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-10-10T01:23:14.9112397Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-10-10T01:23:14.9114088Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-10-10T01:23:14.9115567Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-10-10T01:23:14.9117296Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-10-10T01:23:14.9119819Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-10-10T01:23:14.9121408Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-10-10T01:23:14.9123612Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-10-10T01:23:14.9126047Z * [new branch] gh/williamwen42/281/base -> origin/gh/williamwen42/281/base 2025-10-10T01:23:14.9127845Z * [new branch] gh/williamwen42/281/head -> origin/gh/williamwen42/281/head 2025-10-10T01:23:14.9129523Z * [new branch] gh/williamwen42/281/orig -> origin/gh/williamwen42/281/orig 2025-10-10T01:23:14.9131874Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-10-10T01:23:14.9133777Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-10-10T01:23:14.9135312Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-10-10T01:23:14.9137592Z * [new branch] gh/williamwen42/285/base -> origin/gh/williamwen42/285/base 2025-10-10T01:23:14.9139375Z * [new branch] gh/williamwen42/285/head -> origin/gh/williamwen42/285/head 2025-10-10T01:23:14.9141185Z * [new branch] gh/williamwen42/285/orig -> origin/gh/williamwen42/285/orig 2025-10-10T01:23:14.9143479Z * [new branch] gh/williamwen42/286/base -> origin/gh/williamwen42/286/base 2025-10-10T01:23:14.9145124Z * [new branch] gh/williamwen42/286/head -> origin/gh/williamwen42/286/head 2025-10-10T01:23:14.9146786Z * [new branch] gh/williamwen42/286/orig -> origin/gh/williamwen42/286/orig 2025-10-10T01:23:14.9149165Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-10-10T01:23:14.9150855Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-10-10T01:23:14.9152508Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-10-10T01:23:14.9154894Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-10-10T01:23:14.9156639Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-10-10T01:23:14.9158315Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-10-10T01:23:14.9160986Z * [new branch] gh/williamwen42/289/base -> origin/gh/williamwen42/289/base 2025-10-10T01:23:14.9162736Z * [new branch] gh/williamwen42/289/head -> origin/gh/williamwen42/289/head 2025-10-10T01:23:14.9164467Z * [new branch] gh/williamwen42/289/orig -> origin/gh/williamwen42/289/orig 2025-10-10T01:23:14.9166709Z * [new branch] gh/williamwen42/290/base -> origin/gh/williamwen42/290/base 2025-10-10T01:23:14.9168358Z * [new branch] gh/williamwen42/290/head -> origin/gh/williamwen42/290/head 2025-10-10T01:23:14.9170020Z * [new branch] gh/williamwen42/290/orig -> origin/gh/williamwen42/290/orig 2025-10-10T01:23:14.9172378Z * [new branch] gh/williamwen42/291/base -> origin/gh/williamwen42/291/base 2025-10-10T01:23:14.9174049Z * [new branch] gh/williamwen42/291/head -> origin/gh/williamwen42/291/head 2025-10-10T01:23:14.9175741Z * [new branch] gh/williamwen42/291/orig -> origin/gh/williamwen42/291/orig 2025-10-10T01:23:14.9178063Z * [new branch] gh/williamwen42/292/base -> origin/gh/williamwen42/292/base 2025-10-10T01:23:14.9179793Z * [new branch] gh/williamwen42/292/head -> origin/gh/williamwen42/292/head 2025-10-10T01:23:14.9181932Z * [new branch] gh/williamwen42/292/orig -> origin/gh/williamwen42/292/orig 2025-10-10T01:23:14.9184368Z * [new branch] gh/williamwen42/293/base -> origin/gh/williamwen42/293/base 2025-10-10T01:23:14.9185944Z * [new branch] gh/williamwen42/293/head -> origin/gh/williamwen42/293/head 2025-10-10T01:23:14.9187665Z * [new branch] gh/williamwen42/293/orig -> origin/gh/williamwen42/293/orig 2025-10-10T01:23:14.9189853Z * [new branch] gh/williamwen42/294/base -> origin/gh/williamwen42/294/base 2025-10-10T01:23:14.9191562Z * [new branch] gh/williamwen42/294/head -> origin/gh/williamwen42/294/head 2025-10-10T01:23:14.9193283Z * [new branch] gh/williamwen42/294/orig -> origin/gh/williamwen42/294/orig 2025-10-10T01:23:14.9195542Z * [new branch] gh/williamwen42/295/base -> origin/gh/williamwen42/295/base 2025-10-10T01:23:14.9197629Z * [new branch] gh/williamwen42/295/head -> origin/gh/williamwen42/295/head 2025-10-10T01:23:14.9199608Z * [new branch] gh/williamwen42/295/orig -> origin/gh/williamwen42/295/orig 2025-10-10T01:23:14.9202179Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-10-10T01:23:14.9203861Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-10-10T01:23:14.9205745Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-10-10T01:23:14.9208225Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-10-10T01:23:14.9210019Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-10-10T01:23:14.9211670Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-10-10T01:23:14.9214231Z * [new branch] gh/williamwen42/298/base -> origin/gh/williamwen42/298/base 2025-10-10T01:23:14.9215997Z * [new branch] gh/williamwen42/298/head -> origin/gh/williamwen42/298/head 2025-10-10T01:23:14.9217718Z * [new branch] gh/williamwen42/298/orig -> origin/gh/williamwen42/298/orig 2025-10-10T01:23:14.9220521Z * [new branch] gh/williamwen42/299/base -> origin/gh/williamwen42/299/base 2025-10-10T01:23:14.9222261Z * [new branch] gh/williamwen42/299/head -> origin/gh/williamwen42/299/head 2025-10-10T01:23:14.9223989Z * [new branch] gh/williamwen42/299/orig -> origin/gh/williamwen42/299/orig 2025-10-10T01:23:14.9226411Z * [new branch] gh/williamwen42/300/base -> origin/gh/williamwen42/300/base 2025-10-10T01:23:14.9228064Z * [new branch] gh/williamwen42/300/head -> origin/gh/williamwen42/300/head 2025-10-10T01:23:14.9229797Z * [new branch] gh/williamwen42/300/orig -> origin/gh/williamwen42/300/orig 2025-10-10T01:23:14.9232220Z * [new branch] gh/williamwen42/301/base -> origin/gh/williamwen42/301/base 2025-10-10T01:23:14.9233937Z * [new branch] gh/williamwen42/301/head -> origin/gh/williamwen42/301/head 2025-10-10T01:23:14.9235582Z * [new branch] gh/williamwen42/301/orig -> origin/gh/williamwen42/301/orig 2025-10-10T01:23:14.9237801Z * [new branch] gh/williamwen42/302/base -> origin/gh/williamwen42/302/base 2025-10-10T01:23:14.9239597Z * [new branch] gh/williamwen42/302/head -> origin/gh/williamwen42/302/head 2025-10-10T01:23:14.9241644Z * [new branch] gh/williamwen42/302/orig -> origin/gh/williamwen42/302/orig 2025-10-10T01:23:14.9244088Z * [new branch] gh/williamwen42/303/base -> origin/gh/williamwen42/303/base 2025-10-10T01:23:14.9245768Z * [new branch] gh/williamwen42/303/head -> origin/gh/williamwen42/303/head 2025-10-10T01:23:14.9247497Z * [new branch] gh/williamwen42/303/orig -> origin/gh/williamwen42/303/orig 2025-10-10T01:23:14.9249971Z * [new branch] gh/williamwen42/304/base -> origin/gh/williamwen42/304/base 2025-10-10T01:23:14.9251677Z * [new branch] gh/williamwen42/304/head -> origin/gh/williamwen42/304/head 2025-10-10T01:23:14.9253391Z * [new branch] gh/williamwen42/304/orig -> origin/gh/williamwen42/304/orig 2025-10-10T01:23:14.9255874Z * [new branch] gh/williamwen42/305/base -> origin/gh/williamwen42/305/base 2025-10-10T01:23:14.9257612Z * [new branch] gh/williamwen42/305/head -> origin/gh/williamwen42/305/head 2025-10-10T01:23:14.9259359Z * [new branch] gh/williamwen42/305/orig -> origin/gh/williamwen42/305/orig 2025-10-10T01:23:14.9261564Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-10-10T01:23:14.9263241Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-10-10T01:23:14.9265087Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-10-10T01:23:14.9267421Z * [new branch] gh/williamwen42/307/base -> origin/gh/williamwen42/307/base 2025-10-10T01:23:14.9269201Z * [new branch] gh/williamwen42/307/head -> origin/gh/williamwen42/307/head 2025-10-10T01:23:14.9270952Z * [new branch] gh/williamwen42/307/orig -> origin/gh/williamwen42/307/orig 2025-10-10T01:23:14.9273748Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-10-10T01:23:14.9275491Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-10-10T01:23:14.9277760Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-10-10T01:23:14.9279445Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-10-10T01:23:14.9281752Z * [new branch] gh/xmfan/244/base -> origin/gh/xmfan/244/base 2025-10-10T01:23:14.9283503Z * [new branch] gh/xmfan/244/head -> origin/gh/xmfan/244/head 2025-10-10T01:23:14.9285163Z * [new branch] gh/xmfan/244/orig -> origin/gh/xmfan/244/orig 2025-10-10T01:23:14.9287460Z * [new branch] gh/xmfan/246/base -> origin/gh/xmfan/246/base 2025-10-10T01:23:14.9289171Z * [new branch] gh/xmfan/246/head -> origin/gh/xmfan/246/head 2025-10-10T01:23:14.9290802Z * [new branch] gh/xmfan/246/orig -> origin/gh/xmfan/246/orig 2025-10-10T01:23:14.9292944Z * [new branch] gh/xmfan/253/base -> origin/gh/xmfan/253/base 2025-10-10T01:23:14.9294624Z * [new branch] gh/xmfan/253/head -> origin/gh/xmfan/253/head 2025-10-10T01:23:14.9296386Z * [new branch] gh/xmfan/253/orig -> origin/gh/xmfan/253/orig 2025-10-10T01:23:14.9300128Z * [new branch] gh/xmfan/260/base -> origin/gh/xmfan/260/base 2025-10-10T01:23:14.9301835Z * [new branch] gh/xmfan/260/head -> origin/gh/xmfan/260/head 2025-10-10T01:23:14.9303614Z * [new branch] gh/xmfan/260/orig -> origin/gh/xmfan/260/orig 2025-10-10T01:23:14.9305877Z * [new branch] gh/xmfan/262/base -> origin/gh/xmfan/262/base 2025-10-10T01:23:14.9307515Z * [new branch] gh/xmfan/262/head -> origin/gh/xmfan/262/head 2025-10-10T01:23:14.9309410Z * [new branch] gh/xmfan/262/orig -> origin/gh/xmfan/262/orig 2025-10-10T01:23:14.9311856Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-10-10T01:23:14.9313577Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-10-10T01:23:14.9315231Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-10-10T01:23:14.9317546Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-10-10T01:23:14.9319339Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-10-10T01:23:14.9320990Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-10-10T01:23:14.9323517Z * [new branch] gh/xmfan/281/base -> origin/gh/xmfan/281/base 2025-10-10T01:23:14.9325179Z * [new branch] gh/xmfan/281/head -> origin/gh/xmfan/281/head 2025-10-10T01:23:14.9326869Z * [new branch] gh/xmfan/281/orig -> origin/gh/xmfan/281/orig 2025-10-10T01:23:14.9329218Z * [new branch] gh/xmfan/284/base -> origin/gh/xmfan/284/base 2025-10-10T01:23:14.9330918Z * [new branch] gh/xmfan/284/head -> origin/gh/xmfan/284/head 2025-10-10T01:23:14.9332599Z * [new branch] gh/xmfan/284/orig -> origin/gh/xmfan/284/orig 2025-10-10T01:23:14.9335372Z * [new branch] gh/xmfan/285/base -> origin/gh/xmfan/285/base 2025-10-10T01:23:14.9337220Z * [new branch] gh/xmfan/285/head -> origin/gh/xmfan/285/head 2025-10-10T01:23:14.9339272Z * [new branch] gh/xmfan/285/orig -> origin/gh/xmfan/285/orig 2025-10-10T01:23:14.9341525Z * [new branch] gh/xmfan/286/base -> origin/gh/xmfan/286/base 2025-10-10T01:23:14.9343270Z * [new branch] gh/xmfan/286/head -> origin/gh/xmfan/286/head 2025-10-10T01:23:14.9345022Z * [new branch] gh/xmfan/286/orig -> origin/gh/xmfan/286/orig 2025-10-10T01:23:14.9347482Z * [new branch] gh/xmfan/287/base -> origin/gh/xmfan/287/base 2025-10-10T01:23:14.9349079Z * [new branch] gh/xmfan/287/head -> origin/gh/xmfan/287/head 2025-10-10T01:23:14.9350749Z * [new branch] gh/xmfan/287/orig -> origin/gh/xmfan/287/orig 2025-10-10T01:23:14.9353015Z * [new branch] gh/xmfan/288/base -> origin/gh/xmfan/288/base 2025-10-10T01:23:14.9354850Z * [new branch] gh/xmfan/288/head -> origin/gh/xmfan/288/head 2025-10-10T01:23:14.9356486Z * [new branch] gh/xmfan/288/orig -> origin/gh/xmfan/288/orig 2025-10-10T01:23:14.9358826Z * [new branch] gh/xmfan/289/base -> origin/gh/xmfan/289/base 2025-10-10T01:23:14.9360866Z * [new branch] gh/xmfan/289/head -> origin/gh/xmfan/289/head 2025-10-10T01:23:14.9362433Z * [new branch] gh/xmfan/289/orig -> origin/gh/xmfan/289/orig 2025-10-10T01:23:14.9365253Z * [new branch] gh/xmfan/290/base -> origin/gh/xmfan/290/base 2025-10-10T01:23:14.9367018Z * [new branch] gh/xmfan/290/head -> origin/gh/xmfan/290/head 2025-10-10T01:23:14.9368681Z * [new branch] gh/xmfan/290/orig -> origin/gh/xmfan/290/orig 2025-10-10T01:23:14.9371310Z * [new branch] gh/xmfan/291/base -> origin/gh/xmfan/291/base 2025-10-10T01:23:14.9373118Z * [new branch] gh/xmfan/291/head -> origin/gh/xmfan/291/head 2025-10-10T01:23:14.9374804Z * [new branch] gh/xmfan/291/orig -> origin/gh/xmfan/291/orig 2025-10-10T01:23:14.9377300Z * [new branch] gh/xmfan/292/base -> origin/gh/xmfan/292/base 2025-10-10T01:23:14.9379104Z * [new branch] gh/xmfan/292/head -> origin/gh/xmfan/292/head 2025-10-10T01:23:14.9380841Z * [new branch] gh/xmfan/292/orig -> origin/gh/xmfan/292/orig 2025-10-10T01:23:14.9383157Z * [new branch] gh/xmfan/293/base -> origin/gh/xmfan/293/base 2025-10-10T01:23:14.9385054Z * [new branch] gh/xmfan/293/head -> origin/gh/xmfan/293/head 2025-10-10T01:23:14.9386692Z * [new branch] gh/xmfan/293/orig -> origin/gh/xmfan/293/orig 2025-10-10T01:23:14.9388941Z * [new branch] gh/xmfan/294/base -> origin/gh/xmfan/294/base 2025-10-10T01:23:14.9390636Z * [new branch] gh/xmfan/294/head -> origin/gh/xmfan/294/head 2025-10-10T01:23:14.9392286Z * [new branch] gh/xmfan/294/orig -> origin/gh/xmfan/294/orig 2025-10-10T01:23:14.9394935Z * [new branch] gh/xmfan/295/base -> origin/gh/xmfan/295/base 2025-10-10T01:23:14.9397018Z * [new branch] gh/xmfan/295/head -> origin/gh/xmfan/295/head 2025-10-10T01:23:14.9398766Z * [new branch] gh/xmfan/295/orig -> origin/gh/xmfan/295/orig 2025-10-10T01:23:14.9401227Z * [new branch] gh/xmfan/296/base -> origin/gh/xmfan/296/base 2025-10-10T01:23:14.9402908Z * [new branch] gh/xmfan/296/head -> origin/gh/xmfan/296/head 2025-10-10T01:23:14.9404572Z * [new branch] gh/xmfan/296/orig -> origin/gh/xmfan/296/orig 2025-10-10T01:23:14.9407001Z * [new branch] gh/xmfan/297/base -> origin/gh/xmfan/297/base 2025-10-10T01:23:14.9408901Z * [new branch] gh/xmfan/297/head -> origin/gh/xmfan/297/head 2025-10-10T01:23:14.9410473Z * [new branch] gh/xmfan/297/orig -> origin/gh/xmfan/297/orig 2025-10-10T01:23:14.9412937Z * [new branch] gh/xmfan/298/base -> origin/gh/xmfan/298/base 2025-10-10T01:23:14.9414592Z * [new branch] gh/xmfan/298/head -> origin/gh/xmfan/298/head 2025-10-10T01:23:14.9416249Z * [new branch] gh/xmfan/298/orig -> origin/gh/xmfan/298/orig 2025-10-10T01:23:14.9418661Z * [new branch] gh/xmfan/299/base -> origin/gh/xmfan/299/base 2025-10-10T01:23:14.9420429Z * [new branch] gh/xmfan/299/head -> origin/gh/xmfan/299/head 2025-10-10T01:23:14.9422131Z * [new branch] gh/xmfan/299/orig -> origin/gh/xmfan/299/orig 2025-10-10T01:23:14.9424449Z * [new branch] gh/xmfan/300/base -> origin/gh/xmfan/300/base 2025-10-10T01:23:14.9426130Z * [new branch] gh/xmfan/300/head -> origin/gh/xmfan/300/head 2025-10-10T01:23:14.9427835Z * [new branch] gh/xmfan/300/orig -> origin/gh/xmfan/300/orig 2025-10-10T01:23:14.9430241Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-10-10T01:23:14.9431942Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-10-10T01:23:14.9433589Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-10-10T01:23:14.9436189Z * [new branch] gh/xmfan/302/base -> origin/gh/xmfan/302/base 2025-10-10T01:23:14.9437923Z * [new branch] gh/xmfan/302/head -> origin/gh/xmfan/302/head 2025-10-10T01:23:14.9439773Z * [new branch] gh/xmfan/302/orig -> origin/gh/xmfan/302/orig 2025-10-10T01:23:14.9442174Z * [new branch] gh/xmfan/303/base -> origin/gh/xmfan/303/base 2025-10-10T01:23:14.9443893Z * [new branch] gh/xmfan/303/head -> origin/gh/xmfan/303/head 2025-10-10T01:23:14.9445529Z * [new branch] gh/xmfan/303/orig -> origin/gh/xmfan/303/orig 2025-10-10T01:23:14.9447755Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-10-10T01:23:14.9449426Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-10-10T01:23:14.9451201Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-10-10T01:23:14.9453930Z * [new branch] gh/xuanzhang816/14/base -> origin/gh/xuanzhang816/14/base 2025-10-10T01:23:14.9455615Z * [new branch] gh/xuanzhang816/14/head -> origin/gh/xuanzhang816/14/head 2025-10-10T01:23:14.9457347Z * [new branch] gh/xuanzhang816/14/orig -> origin/gh/xuanzhang816/14/orig 2025-10-10T01:23:14.9459630Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-10-10T01:23:14.9461274Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-10-10T01:23:14.9462956Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-10-10T01:23:14.9465363Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-10-10T01:23:14.9467060Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-10-10T01:23:14.9468726Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-10-10T01:23:14.9471085Z * [new branch] gh/xuanzhang816/25/base -> origin/gh/xuanzhang816/25/base 2025-10-10T01:23:14.9472877Z * [new branch] gh/xuanzhang816/25/head -> origin/gh/xuanzhang816/25/head 2025-10-10T01:23:14.9474623Z * [new branch] gh/xuanzhang816/25/orig -> origin/gh/xuanzhang816/25/orig 2025-10-10T01:23:14.9477039Z * [new branch] gh/xuanzhang816/26/base -> origin/gh/xuanzhang816/26/base 2025-10-10T01:23:14.9478693Z * [new branch] gh/xuanzhang816/26/head -> origin/gh/xuanzhang816/26/head 2025-10-10T01:23:14.9480522Z * [new branch] gh/xuanzhang816/26/orig -> origin/gh/xuanzhang816/26/orig 2025-10-10T01:23:14.9482783Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-10-10T01:23:14.9484415Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-10-10T01:23:14.9486117Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-10-10T01:23:14.9488473Z * [new branch] gh/xuanzhang816/28/base -> origin/gh/xuanzhang816/28/base 2025-10-10T01:23:14.9490605Z * [new branch] gh/xuanzhang816/28/head -> origin/gh/xuanzhang816/28/head 2025-10-10T01:23:14.9492375Z * [new branch] gh/xuanzhang816/28/orig -> origin/gh/xuanzhang816/28/orig 2025-10-10T01:23:14.9494734Z * [new branch] gh/xuanzhang816/29/base -> origin/gh/xuanzhang816/29/base 2025-10-10T01:23:14.9496493Z * [new branch] gh/xuanzhang816/29/head -> origin/gh/xuanzhang816/29/head 2025-10-10T01:23:14.9498337Z * [new branch] gh/xuanzhang816/29/orig -> origin/gh/xuanzhang816/29/orig 2025-10-10T01:23:14.9500602Z * [new branch] gh/xuanzhang816/30/base -> origin/gh/xuanzhang816/30/base 2025-10-10T01:23:14.9502229Z * [new branch] gh/xuanzhang816/30/head -> origin/gh/xuanzhang816/30/head 2025-10-10T01:23:14.9503916Z * [new branch] gh/xuanzhang816/30/orig -> origin/gh/xuanzhang816/30/orig 2025-10-10T01:23:14.9506587Z * [new branch] gh/xuanzhang816/31/base -> origin/gh/xuanzhang816/31/base 2025-10-10T01:23:14.9508275Z * [new branch] gh/xuanzhang816/31/head -> origin/gh/xuanzhang816/31/head 2025-10-10T01:23:14.9509980Z * [new branch] gh/xuanzhang816/31/orig -> origin/gh/xuanzhang816/31/orig 2025-10-10T01:23:14.9512634Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-10-10T01:23:14.9514311Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-10-10T01:23:14.9515960Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-10-10T01:23:14.9518249Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-10-10T01:23:14.9520049Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-10-10T01:23:14.9521694Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-10-10T01:23:14.9524510Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-10-10T01:23:14.9526232Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-10-10T01:23:14.9528437Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-10-10T01:23:14.9530629Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-10-10T01:23:14.9532285Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-10-10T01:23:14.9534050Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-10-10T01:23:14.9536428Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-10-10T01:23:14.9538129Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-10-10T01:23:14.9539850Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-10-10T01:23:14.9542110Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-10-10T01:23:14.9543843Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-10-10T01:23:14.9545616Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-10-10T01:23:14.9547733Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-10-10T01:23:14.9549369Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-10-10T01:23:14.9551188Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-10-10T01:23:14.9553393Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-10-10T01:23:14.9555161Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-10-10T01:23:14.9556874Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-10-10T01:23:14.9559224Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-10-10T01:23:14.9560990Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-10-10T01:23:14.9562571Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-10-10T01:23:14.9565011Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-10-10T01:23:14.9566746Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-10-10T01:23:14.9568433Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-10-10T01:23:14.9570710Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-10-10T01:23:14.9572470Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-10-10T01:23:14.9574723Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-10-10T01:23:14.9576389Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-10-10T01:23:14.9578049Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-10-10T01:23:14.9580965Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-10-10T01:23:14.9582831Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-10-10T01:23:14.9584508Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-10-10T01:23:14.9586754Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-10-10T01:23:14.9588471Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-10-10T01:23:14.9590184Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-10-10T01:23:14.9592785Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-10-10T01:23:14.9594479Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-10-10T01:23:14.9596427Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-10-10T01:23:14.9598746Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-10-10T01:23:14.9600552Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-10-10T01:23:14.9602509Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-10-10T01:23:14.9604922Z * [new branch] gh/yanbing-j/36/base -> origin/gh/yanbing-j/36/base 2025-10-10T01:23:14.9606589Z * [new branch] gh/yanbing-j/36/head -> origin/gh/yanbing-j/36/head 2025-10-10T01:23:14.9608267Z * [new branch] gh/yanbing-j/36/orig -> origin/gh/yanbing-j/36/orig 2025-10-10T01:23:14.9611091Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-10-10T01:23:14.9612815Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-10-10T01:23:14.9614628Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-10-10T01:23:14.9616878Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-10-10T01:23:14.9618536Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-10-10T01:23:14.9620218Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-10-10T01:23:14.9622508Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-10-10T01:23:14.9624173Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-10-10T01:23:14.9625923Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-10-10T01:23:14.9628245Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-10-10T01:23:14.9629989Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-10-10T01:23:14.9631683Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-10-10T01:23:14.9633935Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-10-10T01:23:14.9635688Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-10-10T01:23:14.9637424Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-10-10T01:23:14.9639766Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-10-10T01:23:14.9641460Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-10-10T01:23:14.9643147Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-10-10T01:23:14.9645402Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-10-10T01:23:14.9647085Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-10-10T01:23:14.9648855Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-10-10T01:23:14.9651708Z * [new branch] gh/ydwu4/262/base -> origin/gh/ydwu4/262/base 2025-10-10T01:23:14.9653428Z * [new branch] gh/ydwu4/262/head -> origin/gh/ydwu4/262/head 2025-10-10T01:23:14.9655089Z * [new branch] gh/ydwu4/262/orig -> origin/gh/ydwu4/262/orig 2025-10-10T01:23:14.9657428Z * [new branch] gh/ydwu4/263/base -> origin/gh/ydwu4/263/base 2025-10-10T01:23:14.9659132Z * [new branch] gh/ydwu4/263/head -> origin/gh/ydwu4/263/head 2025-10-10T01:23:14.9660832Z * [new branch] gh/ydwu4/263/orig -> origin/gh/ydwu4/263/orig 2025-10-10T01:23:14.9663155Z * [new branch] gh/ydwu4/269/base -> origin/gh/ydwu4/269/base 2025-10-10T01:23:14.9664948Z * [new branch] gh/ydwu4/269/head -> origin/gh/ydwu4/269/head 2025-10-10T01:23:14.9666661Z * [new branch] gh/ydwu4/269/orig -> origin/gh/ydwu4/269/orig 2025-10-10T01:23:14.9668971Z * [new branch] gh/ydwu4/270/base -> origin/gh/ydwu4/270/base 2025-10-10T01:23:14.9670776Z * [new branch] gh/ydwu4/270/head -> origin/gh/ydwu4/270/head 2025-10-10T01:23:14.9672567Z * [new branch] gh/ydwu4/270/orig -> origin/gh/ydwu4/270/orig 2025-10-10T01:23:14.9674917Z * [new branch] gh/ydwu4/272/base -> origin/gh/ydwu4/272/base 2025-10-10T01:23:14.9676670Z * [new branch] gh/ydwu4/272/head -> origin/gh/ydwu4/272/head 2025-10-10T01:23:14.9678373Z * [new branch] gh/ydwu4/272/orig -> origin/gh/ydwu4/272/orig 2025-10-10T01:23:14.9680818Z * [new branch] gh/ydwu4/275/base -> origin/gh/ydwu4/275/base 2025-10-10T01:23:14.9682535Z * [new branch] gh/ydwu4/275/head -> origin/gh/ydwu4/275/head 2025-10-10T01:23:14.9684117Z * [new branch] gh/ydwu4/275/orig -> origin/gh/ydwu4/275/orig 2025-10-10T01:23:14.9686266Z * [new branch] gh/ydwu4/276/base -> origin/gh/ydwu4/276/base 2025-10-10T01:23:14.9687980Z * [new branch] gh/ydwu4/276/head -> origin/gh/ydwu4/276/head 2025-10-10T01:23:14.9689670Z * [new branch] gh/ydwu4/276/orig -> origin/gh/ydwu4/276/orig 2025-10-10T01:23:14.9692098Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-10-10T01:23:14.9694403Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-10-10T01:23:14.9696222Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-10-10T01:23:14.9700041Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-10-10T01:23:14.9701677Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-10-10T01:23:14.9703322Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-10-10T01:23:14.9705678Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-10-10T01:23:14.9707378Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-10-10T01:23:14.9708998Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-10-10T01:23:14.9711361Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-10-10T01:23:14.9713123Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-10-10T01:23:14.9714943Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-10-10T01:23:14.9717148Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-10-10T01:23:14.9718889Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-10-10T01:23:14.9721316Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-10-10T01:23:14.9723566Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-10-10T01:23:14.9725285Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-10-10T01:23:14.9727044Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-10-10T01:23:14.9729356Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-10-10T01:23:14.9731138Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-10-10T01:23:14.9733131Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-10-10T01:23:14.9735606Z * [new branch] gh/ydwu4/318/base -> origin/gh/ydwu4/318/base 2025-10-10T01:23:14.9737467Z * [new branch] gh/ydwu4/318/head -> origin/gh/ydwu4/318/head 2025-10-10T01:23:14.9739121Z * [new branch] gh/ydwu4/318/orig -> origin/gh/ydwu4/318/orig 2025-10-10T01:23:14.9741328Z * [new branch] gh/ydwu4/319/base -> origin/gh/ydwu4/319/base 2025-10-10T01:23:14.9743070Z * [new branch] gh/ydwu4/319/head -> origin/gh/ydwu4/319/head 2025-10-10T01:23:14.9744787Z * [new branch] gh/ydwu4/319/orig -> origin/gh/ydwu4/319/orig 2025-10-10T01:23:14.9747470Z * [new branch] gh/ydwu4/320/base -> origin/gh/ydwu4/320/base 2025-10-10T01:23:14.9749211Z * [new branch] gh/ydwu4/320/head -> origin/gh/ydwu4/320/head 2025-10-10T01:23:14.9750952Z * [new branch] gh/ydwu4/320/orig -> origin/gh/ydwu4/320/orig 2025-10-10T01:23:14.9753332Z * [new branch] gh/ydwu4/321/base -> origin/gh/ydwu4/321/base 2025-10-10T01:23:14.9755211Z * [new branch] gh/ydwu4/321/head -> origin/gh/ydwu4/321/head 2025-10-10T01:23:14.9756789Z * [new branch] gh/ydwu4/321/orig -> origin/gh/ydwu4/321/orig 2025-10-10T01:23:14.9759209Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-10-10T01:23:14.9761084Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-10-10T01:23:14.9762932Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-10-10T01:23:14.9765286Z * [new branch] gh/ydwu4/324/base -> origin/gh/ydwu4/324/base 2025-10-10T01:23:14.9767098Z * [new branch] gh/ydwu4/324/head -> origin/gh/ydwu4/324/head 2025-10-10T01:23:14.9768791Z * [new branch] gh/ydwu4/324/orig -> origin/gh/ydwu4/324/orig 2025-10-10T01:23:14.9771327Z * [new branch] gh/ydwu4/325/base -> origin/gh/ydwu4/325/base 2025-10-10T01:23:14.9773246Z * [new branch] gh/ydwu4/325/head -> origin/gh/ydwu4/325/head 2025-10-10T01:23:14.9774873Z * [new branch] gh/ydwu4/325/orig -> origin/gh/ydwu4/325/orig 2025-10-10T01:23:14.9777221Z * [new branch] gh/ydwu4/326/base -> origin/gh/ydwu4/326/base 2025-10-10T01:23:14.9779012Z * [new branch] gh/ydwu4/326/head -> origin/gh/ydwu4/326/head 2025-10-10T01:23:14.9780786Z * [new branch] gh/ydwu4/326/orig -> origin/gh/ydwu4/326/orig 2025-10-10T01:23:14.9783327Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-10-10T01:23:14.9785084Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-10-10T01:23:14.9786758Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-10-10T01:23:14.9789255Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-10-10T01:23:14.9791119Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-10-10T01:23:14.9792960Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-10-10T01:23:14.9795401Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-10-10T01:23:14.9797439Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-10-10T01:23:14.9799237Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-10-10T01:23:14.9801923Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-10-10T01:23:14.9803629Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-10-10T01:23:14.9805251Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-10-10T01:23:14.9807479Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-10-10T01:23:14.9809219Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-10-10T01:23:14.9810841Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-10-10T01:23:14.9813166Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-10-10T01:23:14.9814969Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-10-10T01:23:14.9816595Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-10-10T01:23:14.9818822Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-10-10T01:23:14.9820483Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-10-10T01:23:14.9822147Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-10-10T01:23:14.9824406Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-10-10T01:23:14.9826245Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-10-10T01:23:14.9827800Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-10-10T01:23:14.9830038Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-10-10T01:23:14.9831743Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-10-10T01:23:14.9833414Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-10-10T01:23:14.9835744Z * [new branch] gh/ydwu4/336/base -> origin/gh/ydwu4/336/base 2025-10-10T01:23:14.9837528Z * [new branch] gh/ydwu4/336/head -> origin/gh/ydwu4/336/head 2025-10-10T01:23:14.9839270Z * [new branch] gh/ydwu4/336/orig -> origin/gh/ydwu4/336/orig 2025-10-10T01:23:14.9841599Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-10-10T01:23:14.9843312Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-10-10T01:23:14.9845019Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-10-10T01:23:14.9847772Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-10-10T01:23:14.9849493Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-10-10T01:23:14.9852168Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-10-10T01:23:14.9853861Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-10-10T01:23:14.9857107Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-10-10T01:23:14.9859107Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-10-10T01:23:14.9860859Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-10-10T01:23:14.9863234Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-10-10T01:23:14.9864922Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-10-10T01:23:14.9866641Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-10-10T01:23:14.9869579Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-10-10T01:23:14.9871768Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-10-10T01:23:14.9873996Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-10-10T01:23:14.9875611Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-10-10T01:23:14.9878415Z * [new branch] gh/ysiraichi/88/base -> origin/gh/ysiraichi/88/base 2025-10-10T01:23:14.9880341Z * [new branch] gh/ysiraichi/88/head -> origin/gh/ysiraichi/88/head 2025-10-10T01:23:14.9882020Z * [new branch] gh/ysiraichi/88/orig -> origin/gh/ysiraichi/88/orig 2025-10-10T01:23:14.9884867Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2025-10-10T01:23:14.9886548Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2025-10-10T01:23:14.9888278Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2025-10-10T01:23:14.9890834Z * [new branch] gh/zhxchen17/31/base -> origin/gh/zhxchen17/31/base 2025-10-10T01:23:14.9892541Z * [new branch] gh/zhxchen17/31/head -> origin/gh/zhxchen17/31/head 2025-10-10T01:23:14.9894263Z * [new branch] gh/zhxchen17/31/orig -> origin/gh/zhxchen17/31/orig 2025-10-10T01:23:14.9896755Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-10-10T01:23:14.9898485Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-10-10T01:23:14.9900786Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-10-10T01:23:14.9902412Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-10-10T01:23:14.9905582Z * [new branch] gh/zklaus/10/base -> origin/gh/zklaus/10/base 2025-10-10T01:23:14.9907256Z * [new branch] gh/zklaus/10/head -> origin/gh/zklaus/10/head 2025-10-10T01:23:14.9908933Z * [new branch] gh/zklaus/10/orig -> origin/gh/zklaus/10/orig 2025-10-10T01:23:14.9911189Z * [new branch] gh/zklaus/11/base -> origin/gh/zklaus/11/base 2025-10-10T01:23:14.9912913Z * [new branch] gh/zklaus/11/head -> origin/gh/zklaus/11/head 2025-10-10T01:23:14.9914570Z * [new branch] gh/zklaus/11/orig -> origin/gh/zklaus/11/orig 2025-10-10T01:23:14.9916921Z * [new branch] gh/zklaus/15/base -> origin/gh/zklaus/15/base 2025-10-10T01:23:14.9918606Z * [new branch] gh/zklaus/15/head -> origin/gh/zklaus/15/head 2025-10-10T01:23:14.9920617Z * [new branch] gh/zklaus/15/orig -> origin/gh/zklaus/15/orig 2025-10-10T01:23:14.9922918Z * [new branch] gh/zklaus/16/base -> origin/gh/zklaus/16/base 2025-10-10T01:23:14.9924669Z * [new branch] gh/zklaus/16/head -> origin/gh/zklaus/16/head 2025-10-10T01:23:14.9926361Z * [new branch] gh/zklaus/16/orig -> origin/gh/zklaus/16/orig 2025-10-10T01:23:14.9928666Z * [new branch] gh/zklaus/17/base -> origin/gh/zklaus/17/base 2025-10-10T01:23:14.9930351Z * [new branch] gh/zklaus/17/head -> origin/gh/zklaus/17/head 2025-10-10T01:23:14.9932085Z * [new branch] gh/zklaus/17/orig -> origin/gh/zklaus/17/orig 2025-10-10T01:23:14.9934356Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-10-10T01:23:14.9936058Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-10-10T01:23:14.9937721Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-10-10T01:23:14.9939999Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-10-10T01:23:14.9941671Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-10-10T01:23:14.9943469Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-10-10T01:23:14.9945737Z * [new branch] gh/zklaus/7/base -> origin/gh/zklaus/7/base 2025-10-10T01:23:14.9947613Z * [new branch] gh/zklaus/7/head -> origin/gh/zklaus/7/head 2025-10-10T01:23:14.9949277Z * [new branch] gh/zklaus/7/orig -> origin/gh/zklaus/7/orig 2025-10-10T01:23:14.9952164Z * [new branch] gh/zou3519/1177/base -> origin/gh/zou3519/1177/base 2025-10-10T01:23:14.9953880Z * [new branch] gh/zou3519/1177/head -> origin/gh/zou3519/1177/head 2025-10-10T01:23:14.9955629Z * [new branch] gh/zou3519/1177/orig -> origin/gh/zou3519/1177/orig 2025-10-10T01:23:14.9957985Z * [new branch] gh/zou3519/1195/base -> origin/gh/zou3519/1195/base 2025-10-10T01:23:14.9959820Z * [new branch] gh/zou3519/1195/head -> origin/gh/zou3519/1195/head 2025-10-10T01:23:14.9961433Z * [new branch] gh/zou3519/1195/orig -> origin/gh/zou3519/1195/orig 2025-10-10T01:23:14.9963798Z * [new branch] gh/zou3519/1196/base -> origin/gh/zou3519/1196/base 2025-10-10T01:23:14.9965511Z * [new branch] gh/zou3519/1196/head -> origin/gh/zou3519/1196/head 2025-10-10T01:23:14.9967066Z * [new branch] gh/zou3519/1196/orig -> origin/gh/zou3519/1196/orig 2025-10-10T01:23:14.9969735Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-10-10T01:23:14.9971354Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-10-10T01:23:14.9973167Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-10-10T01:23:14.9975291Z * [new branch] gh/zou3519/1198/base -> origin/gh/zou3519/1198/base 2025-10-10T01:23:14.9977501Z * [new branch] gh/zou3519/1198/head -> origin/gh/zou3519/1198/head 2025-10-10T01:23:14.9979687Z * [new branch] gh/zou3519/1198/orig -> origin/gh/zou3519/1198/orig 2025-10-10T01:23:14.9981851Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-10-10T01:23:14.9983607Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-10-10T01:23:14.9985382Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-10-10T01:23:14.9988151Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-10-10T01:23:14.9989797Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-10-10T01:23:14.9991571Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-10-10T01:23:14.9993780Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-10-10T01:23:14.9995487Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-10-10T01:23:14.9997523Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-10-10T01:23:15.0000527Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-10-10T01:23:15.0002599Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-10-10T01:23:15.0005060Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-10-10T01:23:15.0006803Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-10-10T01:23:15.0008964Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-10-10T01:23:15.0011877Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-10-10T01:23:15.0013688Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-10-10T01:23:15.0015435Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-10-10T01:23:15.0017881Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-10-10T01:23:15.0019582Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-10-10T01:23:15.0021487Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-10-10T01:23:15.0024009Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-10-10T01:23:15.0025715Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-10-10T01:23:15.0027412Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-10-10T01:23:15.0029872Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-10-10T01:23:15.0031493Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-10-10T01:23:15.0033225Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-10-10T01:23:15.0035521Z * [new branch] gh/zpcore/16/base -> origin/gh/zpcore/16/base 2025-10-10T01:23:15.0037246Z * [new branch] gh/zpcore/16/head -> origin/gh/zpcore/16/head 2025-10-10T01:23:15.0038970Z * [new branch] gh/zpcore/16/orig -> origin/gh/zpcore/16/orig 2025-10-10T01:23:15.0041627Z * [new branch] gh/zpcore/17/base -> origin/gh/zpcore/17/base 2025-10-10T01:23:15.0043336Z * [new branch] gh/zpcore/17/head -> origin/gh/zpcore/17/head 2025-10-10T01:23:15.0045106Z * [new branch] gh/zpcore/17/orig -> origin/gh/zpcore/17/orig 2025-10-10T01:23:15.0047480Z * [new branch] gh/zpcore/18/base -> origin/gh/zpcore/18/base 2025-10-10T01:23:15.0049128Z * [new branch] gh/zpcore/18/head -> origin/gh/zpcore/18/head 2025-10-10T01:23:15.0050776Z * [new branch] gh/zpcore/18/orig -> origin/gh/zpcore/18/orig 2025-10-10T01:23:15.0053548Z * [new branch] gh/zpcore/19/base -> origin/gh/zpcore/19/base 2025-10-10T01:23:15.0055199Z * [new branch] gh/zpcore/19/head -> origin/gh/zpcore/19/head 2025-10-10T01:23:15.0056950Z * [new branch] gh/zpcore/19/orig -> origin/gh/zpcore/19/orig 2025-10-10T01:23:15.0059270Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-10-10T01:23:15.0061015Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-10-10T01:23:15.0063459Z * [new branch] gh/zpcore/20/base -> origin/gh/zpcore/20/base 2025-10-10T01:23:15.0065204Z * [new branch] gh/zpcore/20/head -> origin/gh/zpcore/20/head 2025-10-10T01:23:15.0066934Z * [new branch] gh/zpcore/20/orig -> origin/gh/zpcore/20/orig 2025-10-10T01:23:15.0069511Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-10-10T01:23:15.0071409Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-10-10T01:23:15.0073237Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-10-10T01:23:15.0075461Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-10-10T01:23:15.0077118Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-10-10T01:23:15.0079359Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-10-10T01:23:15.0081448Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-10-10T01:23:15.0083918Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-10-10T01:23:15.0085681Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-10-10T01:23:15.0087872Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-10-10T01:23:15.0089498Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-10-10T01:23:15.0091632Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-10-10T01:23:15.0093302Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-10-10T01:23:15.0095443Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-10-10T01:23:15.0098699Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-10-10T01:23:15.0100645Z * [new branch] google-main -> origin/google-main 2025-10-10T01:23:15.0102425Z * [new branch] greencontext -> origin/greencontext 2025-10-10T01:23:15.0104806Z * [new branch] guangyey/config -> origin/guangyey/config 2025-10-10T01:23:15.0106479Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-10-10T01:23:15.0108037Z * [new branch] guangyey/reimport -> origin/guangyey/reimport 2025-10-10T01:23:15.0109651Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-10-10T01:23:15.0112308Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-10-10T01:23:15.0114705Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-10-10T01:23:15.0116887Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-10-10T01:23:15.0118617Z * [new branch] hc_baseline -> origin/hc_baseline 2025-10-10T01:23:15.0120649Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-10-10T01:23:15.0122303Z * [new branch] hhh_rand -> origin/hhh_rand 2025-10-10T01:23:15.0124612Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-10-10T01:23:15.0126957Z * [new branch] huba/debug_mode -> origin/huba/debug_mode 2025-10-10T01:23:15.0128796Z * [new branch] huba/dtensor_equal -> origin/huba/dtensor_equal 2025-10-10T01:23:15.0130389Z * [new branch] huba/f1 -> origin/huba/f1 2025-10-10T01:23:15.0132169Z * [new branch] huba/local_tensor -> origin/huba/local_tensor 2025-10-10T01:23:15.0133988Z * [new branch] ideep-update -> origin/ideep-update 2025-10-10T01:23:15.0135897Z * [new branch] increase-asan-build-memory -> origin/increase-asan-build-memory 2025-10-10T01:23:15.0137754Z * [new branch] inductor-perf-increase-timeout -> origin/inductor-perf-increase-timeout 2025-10-10T01:23:15.0139429Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-10-10T01:23:15.0141183Z * [new branch] inline -> origin/inline 2025-10-10T01:23:15.0143031Z * [new branch] inlining -> origin/inlining 2025-10-10T01:23:15.0144987Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-10-10T01:23:15.0146818Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-10-10T01:23:15.0148540Z * [new branch] install_free_tensors -> origin/install_free_tensors 2025-10-10T01:23:15.0150327Z * [new branch] int8_sdpa -> origin/int8_sdpa 2025-10-10T01:23:15.0152151Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-10-10T01:23:15.0154015Z * [new branch] issue#58739 -> origin/issue#58739 2025-10-10T01:23:15.0155882Z * [new branch] issue-161010-dynamo-stride-clone -> origin/issue-161010-dynamo-stride-clone 2025-10-10T01:23:15.0158115Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-10-10T01:23:15.0159779Z * [new branch] jathu/sve -> origin/jathu/sve 2025-10-10T01:23:15.0162418Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-10-10T01:23:15.0164021Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-10-10T01:23:15.0165758Z * [new branch] jeanschmidt-patch-1 -> origin/jeanschmidt-patch-1 2025-10-10T01:23:15.0167654Z * [new branch] jerryzh168-patch-1 -> origin/jerryzh168-patch-1 2025-10-10T01:23:15.0169552Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-10-10T01:23:15.0171464Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-10-10T01:23:15.0173252Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-10-10T01:23:15.0175069Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-10-10T01:23:15.0177489Z * [new branch] justinchu/allowlist-api-onnx -> origin/justinchu/allowlist-api-onnx 2025-10-10T01:23:15.0179062Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-10-10T01:23:15.0180659Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-10-10T01:23:15.0183113Z * [new branch] justinchuby/typo-error -> origin/justinchuby/typo-error 2025-10-10T01:23:15.0185330Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-10-10T01:23:15.0187152Z * [new branch] kainan_test -> origin/kainan_test 2025-10-10T01:23:15.0189492Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-10-10T01:23:15.0191966Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-10-10T01:23:15.0194360Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-10-10T01:23:15.0195996Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-10-10T01:23:15.0197892Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-10-10T01:23:15.0200107Z * [new branch] libtorch_free_so -> origin/libtorch_free_so 2025-10-10T01:23:15.0208085Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-10-10T01:23:15.0208448Z * [new branch] llama4-stable -> origin/llama4-stable 2025-10-10T01:23:15.0208836Z * [new branch] logdetfix -> origin/logdetfix 2025-10-10T01:23:15.0209008Z * [new branch] logsumexp -> origin/logsumexp 2025-10-10T01:23:15.0210541Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-10-10T01:23:15.0213285Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-10-10T01:23:15.0215027Z * [new branch] lucaskabela/cherrypick_163769 -> origin/lucaskabela/cherrypick_163769 2025-10-10T01:23:15.0216587Z * [new branch] lucaskabela/fix_164814 -> origin/lucaskabela/fix_164814 2025-10-10T01:23:15.0218295Z * [new branch] lucaskabela/fix_164823 -> origin/lucaskabela/fix_164823 2025-10-10T01:23:15.0219888Z * [new branch] lucaskabela/fix_164875 -> origin/lucaskabela/fix_164875 2025-10-10T01:23:15.0221503Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-10-10T01:23:15.0223201Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-10-10T01:23:15.0225284Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-10-10T01:23:15.0227454Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-10-10T01:23:15.0229194Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-10-10T01:23:15.0231136Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-10-10T01:23:15.0232640Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-10-10T01:23:15.0234309Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-10-10T01:23:15.0236046Z * [new branch] main -> origin/main 2025-10-10T01:23:15.0238030Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-10-10T01:23:15.0239889Z * [new branch] main-enable-b200-symm-mem-test -> origin/main-enable-b200-symm-mem-test 2025-10-10T01:23:15.0241829Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-10-10T01:23:15.0244394Z * [new branch] malfet-patch-14 -> origin/malfet-patch-14 2025-10-10T01:23:15.0245387Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-10-10T01:23:15.0247226Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-10-10T01:23:15.0249271Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-10-10T01:23:15.0250989Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-10-10T01:23:15.0252787Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-10-10T01:23:15.0254639Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-10-10T01:23:15.0256583Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-10-10T01:23:15.0258878Z * [new branch] malfet-patch-9 -> origin/malfet-patch-9 2025-10-10T01:23:15.0261531Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-10-10T01:23:15.0262847Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-10-10T01:23:15.0265445Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-10-10T01:23:15.0267059Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-10-10T01:23:15.0269351Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-10-10T01:23:15.0271120Z * [new branch] masnesral/pt2_internal_logging -> origin/masnesral/pt2_internal_logging 2025-10-10T01:23:15.0272871Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-10-10T01:23:15.0274627Z * [new branch] mingw_constant_buffer -> origin/mingw_constant_buffer 2025-10-10T01:23:15.0277055Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-10-10T01:23:15.0278640Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-10-10T01:23:15.0280368Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-10-10T01:23:15.0281907Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-10-10T01:23:15.0283582Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-10-10T01:23:15.0285166Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-10-10T01:23:15.0286818Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-10-10T01:23:15.0288794Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-10-10T01:23:15.0290864Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-10-10T01:23:15.0293035Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-10-10T01:23:15.0294650Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-10-10T01:23:15.0296519Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-10-10T01:23:15.0298697Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-10-10T01:23:15.0300477Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-10-10T01:23:15.0302193Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-10-10T01:23:15.0303920Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-10-10T01:23:15.0305811Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-10-10T01:23:15.0307550Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-10-10T01:23:15.0309279Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-10-10T01:23:15.0311209Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-10-10T01:23:15.0312985Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-10-10T01:23:15.0314552Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-10-10T01:23:15.0316295Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-10-10T01:23:15.0317990Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-10-10T01:23:15.0319949Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-10-10T01:23:15.0321981Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-10-10T01:23:15.0323684Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-10-10T01:23:15.0325376Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-10-10T01:23:15.0327160Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-10-10T01:23:15.0328927Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-10-10T01:23:15.0330592Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-10-10T01:23:15.0332370Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-10-10T01:23:15.0334175Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-10-10T01:23:15.0335827Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-10-10T01:23:15.0337534Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-10-10T01:23:15.0339251Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-10-10T01:23:15.0341009Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-10-10T01:23:15.0342735Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-10-10T01:23:15.0344516Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-10-10T01:23:15.0346308Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-10-10T01:23:15.0348121Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-10-10T01:23:15.0349834Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-10-10T01:23:15.0352625Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-10-10T01:23:15.0354229Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-10-10T01:23:15.0355813Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-10-10T01:23:15.0357486Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-10-10T01:23:15.0359225Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-10-10T01:23:15.0360993Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-10-10T01:23:15.0362726Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-10-10T01:23:15.0364361Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-10-10T01:23:15.0366070Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-10-10T01:23:15.0367786Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-10-10T01:23:15.0369461Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-10-10T01:23:15.0371191Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-10-10T01:23:15.0373354Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-10-10T01:23:15.0375183Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-10-10T01:23:15.0376863Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-10-10T01:23:15.0378770Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-10-10T01:23:15.0380415Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-10-10T01:23:15.0382422Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-10-10T01:23:15.0384574Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-10-10T01:23:15.0386014Z * [new branch] mlazos/main -> origin/mlazos/main 2025-10-10T01:23:15.0388041Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-10-10T01:23:15.0389800Z * [new branch] mlazos/mark-static-update -> origin/mlazos/mark-static-update 2025-10-10T01:23:15.0391470Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-10-10T01:23:15.0393187Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-10-10T01:23:15.0394988Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-10-10T01:23:15.0397704Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-10-10T01:23:15.0399582Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-10-10T01:23:15.0401269Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-10-10T01:23:15.0402950Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-10-10T01:23:15.0404775Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-10-10T01:23:15.0406533Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-10-10T01:23:15.0408247Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-10-10T01:23:15.0410077Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-10-10T01:23:15.0411888Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-10-10T01:23:15.0413587Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-10-10T01:23:15.0415442Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-10-10T01:23:15.0417004Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-10-10T01:23:15.0418764Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-10-10T01:23:15.0420556Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-10-10T01:23:15.0422371Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-10-10T01:23:15.0424117Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-10-10T01:23:15.0425790Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-10-10T01:23:15.0427548Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-10-10T01:23:15.0429206Z * [new branch] mlazos/test -> origin/mlazos/test 2025-10-10T01:23:15.0430996Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-10-10T01:23:15.0432863Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-10-10T01:23:15.0434635Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-10-10T01:23:15.0436506Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-10-10T01:23:15.0438248Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-10-10T01:23:15.0440079Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-10-10T01:23:15.0441890Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-10-10T01:23:15.0443787Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-10-10T01:23:15.0445248Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-10-10T01:23:15.0447134Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-10-10T01:23:15.0448606Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-10-10T01:23:15.0450604Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-10-10T01:23:15.0452363Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-10-10T01:23:15.0454049Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-10-10T01:23:15.0455766Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-10-10T01:23:15.0457582Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-10-10T01:23:15.0459365Z * [new branch] module-shim -> origin/module-shim 2025-10-10T01:23:15.0461185Z * [new branch] module-stack -> origin/module-stack 2025-10-10T01:23:15.0462727Z * [new branch] more_ck_Fixes -> origin/more_ck_Fixes 2025-10-10T01:23:15.0464628Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-10-10T01:23:15.0466486Z * [new branch] move_aws_steps_inside_setup_rocm -> origin/move_aws_steps_inside_setup_rocm 2025-10-10T01:23:15.0468241Z * [new branch] msaroufim-patch-1 -> origin/msaroufim-patch-1 2025-10-10T01:23:15.0470659Z * [new branch] msaroufim/be1 -> origin/msaroufim/be1 2025-10-10T01:23:15.0472430Z * [new branch] msaroufim/cn_path -> origin/msaroufim/cn_path 2025-10-10T01:23:15.0473992Z * [new branch] msaroufim/cub -> origin/msaroufim/cub 2025-10-10T01:23:15.0475753Z * [new branch] msaroufim/dtensorfusedadam -> origin/msaroufim/dtensorfusedadam 2025-10-10T01:23:15.0477169Z * [new branch] msaroufim/patchx -> origin/msaroufim/patchx 2025-10-10T01:23:15.0478868Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-10-10T01:23:15.0481361Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-10-10T01:23:15.0483661Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-10-10T01:23:15.0485465Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-10-10T01:23:15.0487920Z * [new branch] nWEIdia/skip-tests-for-pr-159494 -> origin/nWEIdia/skip-tests-for-pr-159494 2025-10-10T01:23:15.0489610Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-10-10T01:23:15.0491379Z * [new branch] new-codegen -> origin/new-codegen 2025-10-10T01:23:15.0493181Z * [new branch] newtest-base -> origin/newtest-base 2025-10-10T01:23:15.0495591Z * [new branch] ngimel/allgather_format -> origin/ngimel/allgather_format 2025-10-10T01:23:15.0499097Z * [new branch] ngimel/cat_perf2 -> origin/ngimel/cat_perf2 2025-10-10T01:23:15.0500587Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-10-10T01:23:15.0502287Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-10-10T01:23:15.0503989Z * [new branch] ngimel/scatter_add_multid -> origin/ngimel/scatter_add_multid 2025-10-10T01:23:15.0505772Z * [new branch] nightly -> origin/nightly 2025-10-10T01:23:15.0508811Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-10-10T01:23:15.0510760Z * [new branch] nikitaved/addmm_epilogue_fusions -> origin/nikitaved/addmm_epilogue_fusions 2025-10-10T01:23:15.0512362Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-10-10T01:23:15.0514027Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-10-10T01:23:15.0515493Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-10-10T01:23:15.0517457Z * [new branch] nmacchioni-patch-10 -> origin/nmacchioni-patch-10 2025-10-10T01:23:15.0519302Z * [new branch] nmacchioni-patch-7 -> origin/nmacchioni-patch-7 2025-10-10T01:23:15.0521234Z * [new branch] nmacchioni-patch-8 -> origin/nmacchioni-patch-8 2025-10-10T01:23:15.0522997Z * [new branch] nmacchioni-patch-9 -> origin/nmacchioni-patch-9 2025-10-10T01:23:15.0524765Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-10-10T01:23:15.0526514Z * [new branch] nofun-hack -> origin/nofun-hack 2025-10-10T01:23:15.0528871Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-10-10T01:23:15.0530513Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-10-10T01:23:15.0533545Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-10-10T01:23:15.0535322Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-10-10T01:23:15.0537176Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-10-10T01:23:15.0539439Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-10-10T01:23:15.0541263Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-10-10T01:23:15.0543179Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-10-10T01:23:15.0544929Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-10-10T01:23:15.0546673Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-10-10T01:23:15.0548428Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-10-10T01:23:15.0550138Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-10-10T01:23:15.0551940Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-10-10T01:23:15.0553623Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-10-10T01:23:15.0555314Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-10-10T01:23:15.0557023Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-10-10T01:23:15.0558671Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-10-10T01:23:15.0560798Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-10-10T01:23:15.0563023Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-10-10T01:23:15.0564679Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-10-10T01:23:15.0568590Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-10-10T01:23:15.0570064Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-10-10T01:23:15.0572980Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-10-10T01:23:15.0574697Z * [new branch] padded-tensor -> origin/padded-tensor 2025-10-10T01:23:15.0576517Z * [new branch] pca2 -> origin/pca2 2025-10-10T01:23:15.0578459Z * [new branch] perf_ops -> origin/perf_ops 2025-10-10T01:23:15.0580190Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-10-10T01:23:15.0582042Z * [new branch] perserve_node_meta_decomp -> origin/perserve_node_meta_decomp 2025-10-10T01:23:15.0583831Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-10-10T01:23:15.0586238Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-10-10T01:23:15.0587806Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-10-10T01:23:15.0589467Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-10-10T01:23:15.0591035Z * [new branch] pianpwk/base_view_shape_key -> origin/pianpwk/base_view_shape_key 2025-10-10T01:23:15.0592724Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-10-10T01:23:15.0594812Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-10-10T01:23:15.0597108Z * [new branch] pianpwk/debug_mode_inductor -> origin/pianpwk/debug_mode_inductor 2025-10-10T01:23:15.0598758Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-10-10T01:23:15.0600834Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-10-10T01:23:15.0602779Z * [new branch] pianpwk/debugmode_show_ids -> origin/pianpwk/debugmode_show_ids 2025-10-10T01:23:15.0604662Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-10-10T01:23:15.0606180Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-10-10T01:23:15.0607933Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-10-10T01:23:15.0609858Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-10-10T01:23:15.0611481Z * [new branch] pianpwk/dtensor_shape_metadata_guard -> origin/pianpwk/dtensor_shape_metadata_guard 2025-10-10T01:23:15.0613264Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-10-10T01:23:15.0615000Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-10-10T01:23:15.0616717Z * [new branch] pianpwk/multi_kernel_l1 -> origin/pianpwk/multi_kernel_l1 2025-10-10T01:23:15.0618426Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-10-10T01:23:15.0620287Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-10-10T01:23:15.0621937Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-10-10T01:23:15.0623769Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-10-10T01:23:15.0625641Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-10-10T01:23:15.0627188Z * [new branch] pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols 2025-10-10T01:23:15.0629003Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-10-10T01:23:15.0630882Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-10-10T01:23:15.0632375Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-10-10T01:23:15.0634319Z * [new branch] pianpwk/triton_benchmark_hints -> origin/pianpwk/triton_benchmark_hints 2025-10-10T01:23:15.0636014Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-10-10T01:23:15.0637653Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-10-10T01:23:15.0639513Z * [new branch] pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last 2025-10-10T01:23:15.0641280Z * [new branch] pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2 2025-10-10T01:23:15.0642932Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-10-10T01:23:15.0644647Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-10-10T01:23:15.0646392Z * [new branch] pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer 2025-10-10T01:23:15.0648690Z * [new branch] piz/add_wait -> origin/piz/add_wait 2025-10-10T01:23:15.0650422Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-10-10T01:23:15.0652179Z * [new branch] pool-separate -> origin/pool-separate 2025-10-10T01:23:15.0653939Z * [new branch] pr-156087 -> origin/pr-156087 2025-10-10T01:23:15.0656400Z * [new branch] pr/131860 -> origin/pr/131860 2025-10-10T01:23:15.0658261Z * [new branch] pre_compile_checks -> origin/pre_compile_checks 2025-10-10T01:23:15.0660015Z * [new branch] predispatch_to -> origin/predispatch_to 2025-10-10T01:23:15.0661969Z * [new branch] prepare-perf-baseline-number-2.8 -> origin/prepare-perf-baseline-number-2.8 2025-10-10T01:23:15.0663478Z * [new branch] prepare-perf-number-2.9 -> origin/prepare-perf-number-2.9 2025-10-10T01:23:15.0665361Z * [new branch] profiler-enabled -> origin/profiler-enabled 2025-10-10T01:23:15.0667099Z * [new branch] provenance_doc_2 -> origin/provenance_doc_2 2025-10-10T01:23:15.0668994Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-10-10T01:23:15.0670741Z * [new branch] pyobjectslot -> origin/pyobjectslot 2025-10-10T01:23:15.0672816Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-10-10T01:23:15.0675637Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-10-10T01:23:15.0678112Z * [new branch] quantile-docs -> origin/quantile-docs 2025-10-10T01:23:15.0679919Z * [new branch] quint-bits -> origin/quint-bits 2025-10-10T01:23:15.0681362Z * [new branch] reland-fx-annotate -> origin/reland-fx-annotate 2025-10-10T01:23:15.0683279Z * [new branch] reland_req_nvsh -> origin/reland_req_nvsh 2025-10-10T01:23:15.0685687Z * [new branch] release/1.10 -> origin/release/1.10 2025-10-10T01:23:15.0687552Z * [new branch] release/1.11 -> origin/release/1.11 2025-10-10T01:23:15.0689305Z * [new branch] release/1.12 -> origin/release/1.12 2025-10-10T01:23:15.0691182Z * [new branch] release/1.13 -> origin/release/1.13 2025-10-10T01:23:15.0692656Z * [new branch] release/1.4 -> origin/release/1.4 2025-10-10T01:23:15.0694166Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-10-10T01:23:15.0696615Z * [new branch] release/1.5 -> origin/release/1.5 2025-10-10T01:23:15.0698650Z * [new branch] release/1.6 -> origin/release/1.6 2025-10-10T01:23:15.0700298Z * [new branch] release/1.7 -> origin/release/1.7 2025-10-10T01:23:15.0702174Z * [new branch] release/1.8 -> origin/release/1.8 2025-10-10T01:23:15.0704151Z * [new branch] release/1.9 -> origin/release/1.9 2025-10-10T01:23:15.0705800Z * [new branch] release/2.0 -> origin/release/2.0 2025-10-10T01:23:15.0707530Z * [new branch] release/2.1 -> origin/release/2.1 2025-10-10T01:23:15.0709755Z * [new branch] release/2.2 -> origin/release/2.2 2025-10-10T01:23:15.0711841Z * [new branch] release/2.3 -> origin/release/2.3 2025-10-10T01:23:15.0714048Z * [new branch] release/2.4 -> origin/release/2.4 2025-10-10T01:23:15.0716157Z * [new branch] release/2.5 -> origin/release/2.5 2025-10-10T01:23:15.0717948Z * [new branch] release/2.6 -> origin/release/2.6 2025-10-10T01:23:15.0719881Z * [new branch] release/2.7 -> origin/release/2.7 2025-10-10T01:23:15.0721754Z * [new branch] release/2.8 -> origin/release/2.8 2025-10-10T01:23:15.0723549Z * [new branch] release/2.9 -> origin/release/2.9 2025-10-10T01:23:15.0725386Z * [new branch] release_notes -> origin/release_notes 2025-10-10T01:23:15.0727271Z * [new branch] remove_header_code -> origin/remove_header_code 2025-10-10T01:23:15.0729452Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-10-10T01:23:15.0731248Z * [new branch] repackage-vllm-nightlies -> origin/repackage-vllm-nightlies 2025-10-10T01:23:15.0733146Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-10-10T01:23:15.0734752Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-10-10T01:23:15.0736444Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-10-10T01:23:15.0738294Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-10-10T01:23:15.0741702Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-10-10T01:23:15.0744838Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-10-10T01:23:15.0748148Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-10-10T01:23:15.0750221Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-10-10T01:23:15.0752968Z * [new branch] revert-163802-camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 -> origin/revert-163802-camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 2025-10-10T01:23:15.0754181Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-10-10T01:23:15.0756022Z * [new branch] rocm-test-yml-update -> origin/rocm-test-yml-update 2025-10-10T01:23:15.0757790Z * [new branch] rocm_op_bench -> origin/rocm_op_bench 2025-10-10T01:23:15.0760348Z * [new branch] ruisi/aot_eager_pass -> origin/ruisi/aot_eager_pass 2025-10-10T01:23:15.0761851Z * [new branch] ruisi/placement_trace -> origin/ruisi/placement_trace 2025-10-10T01:23:15.0764595Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-10-10T01:23:15.0765961Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-10-10T01:23:15.0768438Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-10-10T01:23:15.0769873Z * [new branch] rzou/njt -> origin/rzou/njt 2025-10-10T01:23:15.0771792Z * [new branch] rzou/pca -> origin/rzou/pca 2025-10-10T01:23:15.0773170Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-10-10T01:23:15.0774915Z * [new branch] rzou/setup_context -> origin/rzou/setup_context 2025-10-10T01:23:15.0776750Z * [new branch] samplevllm -> origin/samplevllm 2025-10-10T01:23:15.0779660Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-10-10T01:23:15.0781090Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-10-10T01:23:15.0782865Z * [new branch] save -> origin/save 2025-10-10T01:23:15.0785206Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-10-10T01:23:15.0787130Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-10-10T01:23:15.0789339Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-10-10T01:23:15.0791160Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-10-10T01:23:15.0793055Z * [new branch] shoumikhin-patch-12 -> origin/shoumikhin-patch-12 2025-10-10T01:23:15.0794854Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-10-10T01:23:15.0797549Z * [new branch] soulitzer/reland-codev-grad-dtype -> origin/soulitzer/reland-codev-grad-dtype 2025-10-10T01:23:15.0798937Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-10-10T01:23:15.0801560Z * [new branch] sqzhang/flight4plus -> origin/sqzhang/flight4plus 2025-10-10T01:23:15.0803837Z * [new branch] sraikund16/test -> origin/sraikund16/test 2025-10-10T01:23:15.0805751Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-10-10T01:23:15.0807513Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-10-10T01:23:15.0809290Z * [new branch] suo -> origin/suo 2025-10-10T01:23:15.0811202Z * [new branch] support-uv-in-collect_env -> origin/support-uv-in-collect_env 2025-10-10T01:23:15.0812998Z * [new branch] sve-poc -> origin/sve-poc 2025-10-10T01:23:15.0814745Z * [new branch] svekars-patch-1 -> origin/svekars-patch-1 2025-10-10T01:23:15.0816518Z * [new branch] svekars-patch-2 -> origin/svekars-patch-2 2025-10-10T01:23:15.0818469Z * [new branch] svekars-patch-3 -> origin/svekars-patch-3 2025-10-10T01:23:15.0820279Z * [new branch] svekars-patch-4 -> origin/svekars-patch-4 2025-10-10T01:23:15.0822116Z * [new branch] svekars-patch-5 -> origin/svekars-patch-5 2025-10-10T01:23:15.0823961Z * [new branch] switch-bn -> origin/switch-bn 2025-10-10T01:23:15.0825737Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-10-10T01:23:15.0828507Z * [new branch] tenpercent/ck_rocm_ci_v3 -> origin/tenpercent/ck_rocm_ci_v3 2025-10-10T01:23:15.0830334Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-10-10T01:23:15.0832222Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-10-10T01:23:15.0834117Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-10-10T01:23:15.0835596Z * [new branch] test-old -> origin/test-old 2025-10-10T01:23:15.0837657Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-10-10T01:23:15.0840100Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-10-10T01:23:15.0841479Z * [new branch] test/inductor -> origin/test/inductor 2025-10-10T01:23:15.0843375Z * [new branch] test_quantization -> origin/test_quantization 2025-10-10T01:23:15.0845884Z * [new branch] tianren/customOp_autotune -> origin/tianren/customOp_autotune 2025-10-10T01:23:15.0847410Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-10-10T01:23:15.0848883Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-10-10T01:23:15.0850707Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-10-10T01:23:15.0852256Z * [new branch] tianren/remove_repeate -> origin/tianren/remove_repeate 2025-10-10T01:23:15.0854136Z * [new branch] tianren/test -> origin/tianren/test 2025-10-10T01:23:15.0855922Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-10-10T01:23:15.0857642Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-10-10T01:23:15.0859379Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-10-10T01:23:15.0861204Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-10-10T01:23:15.0862923Z * [new branch] transpose_pack_fusion -> origin/transpose_pack_fusion 2025-10-10T01:23:15.0864656Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-10-10T01:23:15.0866463Z * [new branch] triton_kernel -> origin/triton_kernel 2025-10-10T01:23:15.0868353Z * [new branch] trunk-tagging-multi-commits -> origin/trunk-tagging-multi-commits 2025-10-10T01:23:15.0870024Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-10-10T01:23:15.0871961Z * [new branch] type_dec -> origin/type_dec 2025-10-10T01:23:15.0873914Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-10-10T01:23:15.0875630Z * [new branch] unlift -> origin/unlift 2025-10-10T01:23:15.0878185Z * [new branch] update-audio-commit-hash/17567864209-1799-1 -> origin/update-audio-commit-hash/17567864209-1799-1 2025-10-10T01:23:15.0879776Z * [new branch] update-audio-commit-hash/17599208654-1801-1 -> origin/update-audio-commit-hash/17599208654-1801-1 2025-10-10T01:23:15.0881473Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-10-10T01:23:15.0883026Z * [new branch] update-audio-commit-hash/17657093113-1804-1 -> origin/update-audio-commit-hash/17657093113-1804-1 2025-10-10T01:23:15.0884700Z * [new branch] update-audio-commit-hash/17688961747-1806-1 -> origin/update-audio-commit-hash/17688961747-1806-1 2025-10-10T01:23:15.0886757Z * [new branch] update-audio-commit-hash/17703952853-1807-1 -> origin/update-audio-commit-hash/17703952853-1807-1 2025-10-10T01:23:15.0888860Z * [new branch] update-audio-commit-hash/18392707270-1874-1 -> origin/update-audio-commit-hash/18392707270-1874-1 2025-10-10T01:23:15.0890430Z * [new branch] update-dynamic-shapes-doc -> origin/update-dynamic-shapes-doc 2025-10-10T01:23:15.0893112Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-10-10T01:23:15.0895369Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-10-10T01:23:15.0898237Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-10-10T01:23:15.0899502Z * [new branch] update-vision-commit-hash/18361653903-1869-1 -> origin/update-vision-commit-hash/18361653903-1869-1 2025-10-10T01:23:15.0901898Z * [new branch] update-vllm-commit-hash/17536029887-1798-1 -> origin/update-vllm-commit-hash/17536029887-1798-1 2025-10-10T01:23:15.0903530Z * [new branch] update-vllm-commit-hash/17599208654-1801-1 -> origin/update-vllm-commit-hash/17599208654-1801-1 2025-10-10T01:23:15.0905184Z * [new branch] update-vllm-commit-hash/17657093113-1804-1 -> origin/update-vllm-commit-hash/17657093113-1804-1 2025-10-10T01:23:15.0906777Z * [new branch] update-vllm-commit-hash/17703952853-1807-1 -> origin/update-vllm-commit-hash/17703952853-1807-1 2025-10-10T01:23:15.0908393Z * [new branch] update-vllm-commit-hash/17718740812-1808-1 -> origin/update-vllm-commit-hash/17718740812-1808-1 2025-10-10T01:23:15.0910482Z * [new branch] update-vllm-commit-hash/17782703922-1813-1 -> origin/update-vllm-commit-hash/17782703922-1813-1 2025-10-10T01:23:15.0912479Z * [new branch] update-vllm-commit-hash/17814169036-1822-1 -> origin/update-vllm-commit-hash/17814169036-1822-1 2025-10-10T01:23:15.0914996Z * [new branch] update-vllm-commit-hash/17844794719-1823-1 -> origin/update-vllm-commit-hash/17844794719-1823-1 2025-10-10T01:23:15.0916598Z * [new branch] update-vllm-commit-hash/17872674059-1830-1 -> origin/update-vllm-commit-hash/17872674059-1830-1 2025-10-10T01:23:15.0918643Z * [new branch] update-vllm-commit-hash/17901034819-1833-1 -> origin/update-vllm-commit-hash/17901034819-1833-1 2025-10-10T01:23:15.0920296Z * [new branch] update-vllm-commit-hash/17932176396-1836-1 -> origin/update-vllm-commit-hash/17932176396-1836-1 2025-10-10T01:23:15.0922176Z * [new branch] update-vllm-commit-hash/17962545886-1842-1 -> origin/update-vllm-commit-hash/17962545886-1842-1 2025-10-10T01:23:15.0923873Z * [new branch] update-vllm-commit-hash/17993166855-1844-1 -> origin/update-vllm-commit-hash/17993166855-1844-1 2025-10-10T01:23:15.0925594Z * [new branch] update-vllm-commit-hash/18052321282-1848-1 -> origin/update-vllm-commit-hash/18052321282-1848-1 2025-10-10T01:23:15.0927275Z * [new branch] update-vllm-commit-hash/18066820738-1849-1 -> origin/update-vllm-commit-hash/18066820738-1849-1 2025-10-10T01:23:15.0928997Z * [new branch] update-vllm-commit-hash/18081987460-1850-1 -> origin/update-vllm-commit-hash/18081987460-1850-1 2025-10-10T01:23:15.0930697Z * [new branch] update-vllm-commit-hash/18114584510-1852-1 -> origin/update-vllm-commit-hash/18114584510-1852-1 2025-10-10T01:23:15.0932663Z * [new branch] update-vllm-commit-hash/18147226974-1853-1 -> origin/update-vllm-commit-hash/18147226974-1853-1 2025-10-10T01:23:15.0934323Z * [new branch] update-vllm-commit-hash/18236802781-1857-1 -> origin/update-vllm-commit-hash/18236802781-1857-1 2025-10-10T01:23:15.0936701Z * [new branch] update-xla-commit-hash/17725712604-203-1 -> origin/update-xla-commit-hash/17725712604-203-1 2025-10-10T01:23:15.0938732Z * [new branch] update-xla-commit-hash/17908176340-204-1 -> origin/update-xla-commit-hash/17908176340-204-1 2025-10-10T01:23:15.0940264Z * [new branch] update-xla-commit-hash/18273597034-206-1 -> origin/update-xla-commit-hash/18273597034-206-1 2025-10-10T01:23:15.0942153Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-10-10T01:23:15.0943703Z * [new branch] update_executorch_pin -> origin/update_executorch_pin 2025-10-10T01:23:15.0945747Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-10-10T01:23:15.0947479Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-10-10T01:23:15.0949771Z * [new branch] update_slow_tests_1757922057 -> origin/update_slow_tests_1757922057 2025-10-10T01:23:15.0951531Z * [new branch] update_slow_tests_1758526845 -> origin/update_slow_tests_1758526845 2025-10-10T01:23:15.0953188Z * [new branch] update_slow_tests_1759736444 -> origin/update_slow_tests_1759736444 2025-10-10T01:23:15.0954932Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-10-10T01:23:15.0956664Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-10-10T01:23:15.0958436Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-10-10T01:23:15.0960528Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-10-10T01:23:15.0962445Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-10-10T01:23:15.0964621Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-10-10T01:23:15.0966483Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-10-10T01:23:15.0968365Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-10-10T01:23:15.0970203Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-10-10T01:23:15.0972025Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-10-10T01:23:15.0973861Z * [new branch] validate_fn -> origin/validate_fn 2025-10-10T01:23:15.0975971Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-10-10T01:23:15.0977771Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-10-10T01:23:15.0979534Z * [new branch] varlen-api -> origin/varlen-api 2025-10-10T01:23:15.0981266Z * [new branch] varlen_api -> origin/varlen_api 2025-10-10T01:23:15.0983705Z * [new branch] viable/strict -> origin/viable/strict 2025-10-10T01:23:15.0986239Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-10-10T01:23:15.0987715Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-10-10T01:23:15.0989622Z * [new branch] vllmpin -> origin/vllmpin 2025-10-10T01:23:15.0991984Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-10-10T01:23:15.0994320Z * [new branch] whc/flight51 -> origin/whc/flight51 2025-10-10T01:23:15.0996013Z * [new branch] whc/flight53 -> origin/whc/flight53 2025-10-10T01:23:15.1000095Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-10-10T01:23:15.1001673Z * [new branch] whc/uneven -> origin/whc/uneven 2025-10-10T01:23:15.1003778Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-10-10T01:23:15.1005454Z * [new branch] williamwen42-patch-1 -> origin/williamwen42-patch-1 2025-10-10T01:23:15.1007270Z * [new branch] win_warnings -> origin/win_warnings 2025-10-10T01:23:15.1009287Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-10-10T01:23:15.1011269Z * [new branch] windows_mmap -> origin/windows_mmap 2025-10-10T01:23:15.1012948Z * [new branch] xmfan-war -> origin/xmfan-war 2025-10-10T01:23:15.1015207Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-10-10T01:23:15.1016887Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-10-10T01:23:15.1018579Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-10-10T01:23:15.1020074Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-10-10T01:23:15.1021669Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-10-10T01:23:15.1023727Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-10-10T01:23:15.1025278Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-10-10T01:23:15.1027495Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-10-10T01:23:15.1029835Z * [new branch] xmfan/ca_cudagraphs -> origin/xmfan/ca_cudagraphs 2025-10-10T01:23:15.1031700Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-10-10T01:23:15.1033226Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-10-10T01:23:15.1035194Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-10-10T01:23:15.1036836Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-10-10T01:23:15.1038371Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-10-10T01:23:15.1040246Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-10-10T01:23:15.1041874Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-10-10T01:23:15.1043695Z * [new branch] xmfan/ca_mem_base -> origin/xmfan/ca_mem_base 2025-10-10T01:23:15.1045347Z * [new branch] xmfan/ca_mem_fix -> origin/xmfan/ca_mem_fix 2025-10-10T01:23:15.1047099Z * [new branch] xmfan/ca_move_to_cuda -> origin/xmfan/ca_move_to_cuda 2025-10-10T01:23:15.1048790Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-10-10T01:23:15.1050540Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-10-10T01:23:15.1052377Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-10-10T01:23:15.1054462Z * [new branch] xmfan/ca_scalar -> origin/xmfan/ca_scalar 2025-10-10T01:23:15.1056269Z * [new branch] xmfan/ca_subclass_mem_fix -> origin/xmfan/ca_subclass_mem_fix 2025-10-10T01:23:15.1057970Z * [new branch] xmfan/ca_warm_mem -> origin/xmfan/ca_warm_mem 2025-10-10T01:23:15.1059721Z * [new branch] xmfan/ca_warm_mem_base -> origin/xmfan/ca_warm_mem_base 2025-10-10T01:23:15.1061392Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-10-10T01:23:15.1063143Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-10-10T01:23:15.1064891Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-10-10T01:23:15.1066581Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-10-10T01:23:15.1068351Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-10-10T01:23:15.1070157Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-10-10T01:23:15.1071930Z * [new branch] xmfan/issue_123374 -> origin/xmfan/issue_123374 2025-10-10T01:23:15.1073952Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-10-10T01:23:15.1075628Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-10-10T01:23:15.1077097Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-10-10T01:23:15.1078948Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-10-10T01:23:15.1080882Z * [new branch] xmfan/test -> origin/xmfan/test 2025-10-10T01:23:15.1083351Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-10-10T01:23:15.1084719Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-10-10T01:23:15.1086527Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-10-10T01:23:15.1088263Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-10-10T01:23:15.1090640Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-10-10T01:23:15.1092461Z * [new branch] yiming/improve_sharding_error_msg -> origin/yiming/improve_sharding_error_msg 2025-10-10T01:23:15.1093915Z * [new branch] yiming/precompile_benchmark -> origin/yiming/precompile_benchmark 2025-10-10T01:23:15.1095826Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-10-10T01:23:15.1098650Z * [new branch] ysiraichi/install-fmtlib-headers-v12 -> origin/ysiraichi/install-fmtlib-headers-v12 2025-10-10T01:23:15.1100733Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-10-10T01:23:15.1103044Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-10-10T01:23:15.1104478Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-10-10T01:23:15.1106151Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-10-10T01:23:15.1107827Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-10-10T01:23:15.1109706Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-10-10T01:23:15.1111397Z * [new branch] zb2p -> origin/zb2p 2025-10-10T01:23:15.1113247Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-10-10T01:23:15.1115870Z * [new branch] zhxchen17/aot_compile_fix_load_guard_manager -> origin/zhxchen17/aot_compile_fix_load_guard_manager 2025-10-10T01:23:15.1118068Z * [new branch] zhxchen17/precompile/source_info -> origin/zhxchen17/precompile/source_info 2025-10-10T01:23:15.1120278Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-10-10T01:23:15.1122699Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-10-10T01:23:15.1125034Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-10-10T01:23:15.1126682Z * [new branch] zxiiro/c7i-docs -> origin/zxiiro/c7i-docs 2025-10-10T01:23:15.1128436Z * [new branch] zxiiro/c7i-linux-4xlarge -> origin/zxiiro/c7i-linux-4xlarge 2025-10-10T01:23:15.1130198Z * [new branch] zxiiro/c7i-linux-build-yaml -> origin/zxiiro/c7i-linux-build-yaml 2025-10-10T01:23:15.1131890Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-10-10T01:23:15.1133656Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-10-10T01:23:15.1135178Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-10-10T01:23:15.1136608Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-10-10T01:23:15.1138942Z * [new tag] ciflow/b200-symm-mem/163767 -> ciflow/b200-symm-mem/163767 2025-10-10T01:23:15.1140684Z * [new tag] ciflow/b200/163955 -> ciflow/b200/163955 2025-10-10T01:23:15.1141980Z * [new tag] ciflow/binaries/157432 -> ciflow/binaries/157432 2025-10-10T01:23:15.1143134Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-10-10T01:23:15.1144601Z * [new tag] ciflow/binaries/164769 -> ciflow/binaries/164769 2025-10-10T01:23:15.1145864Z * [new tag] ciflow/binaries/164894 -> ciflow/binaries/164894 2025-10-10T01:23:15.1147167Z * [new tag] ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432 2025-10-10T01:23:15.1148480Z * [new tag] ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432 2025-10-10T01:23:15.1149740Z * [new tag] ciflow/binaries_wheel/159104 -> ciflow/binaries_wheel/159104 2025-10-10T01:23:15.1151098Z * [new tag] ciflow/binaries_wheel/164935 -> ciflow/binaries_wheel/164935 2025-10-10T01:23:15.1152540Z * [new tag] ciflow/h100-cutlass-backend/163767 -> ciflow/h100-cutlass-backend/163767 2025-10-10T01:23:15.1153657Z * [new tag] ciflow/h100-cutlass-backend/164747 -> ciflow/h100-cutlass-backend/164747 2025-10-10T01:23:15.1155019Z * [new tag] ciflow/h100-distributed/163767 -> ciflow/h100-distributed/163767 2025-10-10T01:23:15.1156494Z * [new tag] ciflow/h100-symm-mem/151845 -> ciflow/h100-symm-mem/151845 2025-10-10T01:23:15.1157654Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-10-10T01:23:15.1158831Z * [new tag] ciflow/h100-symm-mem/163767 -> ciflow/h100-symm-mem/163767 2025-10-10T01:23:15.1160115Z * [new tag] ciflow/h100-symm-mem/164747 -> ciflow/h100-symm-mem/164747 2025-10-10T01:23:15.1161248Z * [new tag] ciflow/h100-symm-mem/164965 -> ciflow/h100-symm-mem/164965 2025-10-10T01:23:15.1162357Z * [new tag] ciflow/h100-symm-mem/165101 -> ciflow/h100-symm-mem/165101 2025-10-10T01:23:15.1163839Z * [new tag] ciflow/h100/163955 -> ciflow/h100/163955 2025-10-10T01:23:15.1164880Z * [new tag] ciflow/h100/164474 -> ciflow/h100/164474 2025-10-10T01:23:15.1166831Z * [new tag] ciflow/h100/164705 -> ciflow/h100/164705 2025-10-10T01:23:15.1167898Z * [new tag] ciflow/h100/164790 -> ciflow/h100/164790 2025-10-10T01:23:15.1169144Z * [new tag] ciflow/h100/164930 -> ciflow/h100/164930 2025-10-10T01:23:15.1170216Z * [new tag] ciflow/h100/165055 -> ciflow/h100/165055 2025-10-10T01:23:15.1172129Z * [new tag] ciflow/inductor-micro-benchmark/164747 -> ciflow/inductor-micro-benchmark/164747 2025-10-10T01:23:15.1173484Z * [new tag] ciflow/inductor-perf-compare/163767 -> ciflow/inductor-perf-compare/163767 2025-10-10T01:23:15.1174563Z * [new tag] ciflow/inductor-perf-compare/164747 -> ciflow/inductor-perf-compare/164747 2025-10-10T01:23:15.1176293Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/151845 -> ciflow/inductor-perf-test-nightly-rocm/151845 2025-10-10T01:23:15.1177346Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/164747 -> ciflow/inductor-perf-test-nightly-rocm/164747 2025-10-10T01:23:15.1178965Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/161512 -> ciflow/inductor-perf-test-nightly-x86-zen/161512 2025-10-10T01:23:15.1180039Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/162954 -> ciflow/inductor-perf-test-nightly-x86-zen/162954 2025-10-10T01:23:15.1181139Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/163767 -> ciflow/inductor-perf-test-nightly-x86-zen/163767 2025-10-10T01:23:15.1182330Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/164126 -> ciflow/inductor-perf-test-nightly-x86-zen/164126 2025-10-10T01:23:15.1183491Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/164747 -> ciflow/inductor-perf-test-nightly-x86-zen/164747 2025-10-10T01:23:15.1185309Z * [new tag] ciflow/inductor-periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 -> ciflow/inductor-periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 2025-10-10T01:23:15.1186299Z * [new tag] ciflow/inductor-periodic/156592 -> ciflow/inductor-periodic/156592 2025-10-10T01:23:15.1187364Z * [new tag] ciflow/inductor-periodic/164492 -> ciflow/inductor-periodic/164492 2025-10-10T01:23:15.1189251Z * [new tag] ciflow/inductor-periodic/73adac05d13babb75410c3e033fdce57aa16881a -> ciflow/inductor-periodic/73adac05d13babb75410c3e033fdce57aa16881a 2025-10-10T01:23:15.1190101Z * [new tag] ciflow/inductor-rocm/151845 -> ciflow/inductor-rocm/151845 2025-10-10T01:23:15.1191533Z * [new tag] ciflow/inductor-rocm/161280 -> ciflow/inductor-rocm/161280 2025-10-10T01:23:15.1192973Z * [new tag] ciflow/inductor-rocm/162478 -> ciflow/inductor-rocm/162478 2025-10-10T01:23:15.1193874Z * [new tag] ciflow/inductor-rocm/163767 -> ciflow/inductor-rocm/163767 2025-10-10T01:23:15.1195329Z * [new tag] ciflow/inductor-rocm/164618 -> ciflow/inductor-rocm/164618 2025-10-10T01:23:15.1196465Z * [new tag] ciflow/inductor-rocm/164747 -> ciflow/inductor-rocm/164747 2025-10-10T01:23:15.1197848Z * [new tag] ciflow/inductor-rocm/164769 -> ciflow/inductor-rocm/164769 2025-10-10T01:23:15.1198982Z * [new tag] ciflow/inductor-rocm/165080 -> ciflow/inductor-rocm/165080 2025-10-10T01:23:15.1200527Z * [new tag] ciflow/inductor-rocm/165115 -> ciflow/inductor-rocm/165115 2025-10-10T01:23:15.1201882Z * [new tag] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-10-10T01:23:15.1202933Z * [new tag] ciflow/inductor/148180 -> ciflow/inductor/148180 2025-10-10T01:23:15.1204066Z * [new tag] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-10-10T01:23:15.1205407Z * [new tag] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-10-10T01:23:15.1206375Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-10-10T01:23:15.1207512Z * [new tag] ciflow/inductor/149003 -> ciflow/inductor/149003 2025-10-10T01:23:15.1208633Z * [new tag] ciflow/inductor/151845 -> ciflow/inductor/151845 2025-10-10T01:23:15.1209771Z * [new tag] ciflow/inductor/152624 -> ciflow/inductor/152624 2025-10-10T01:23:15.1210910Z * [new tag] ciflow/inductor/156592 -> ciflow/inductor/156592 2025-10-10T01:23:15.1212176Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-10-10T01:23:15.1213320Z * [new tag] ciflow/inductor/157743 -> ciflow/inductor/157743 2025-10-10T01:23:15.1215095Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-10-10T01:23:15.1216710Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-10-10T01:23:15.1218213Z * [new tag] ciflow/inductor/158872 -> ciflow/inductor/158872 2025-10-10T01:23:15.1219652Z * [new tag] ciflow/inductor/158932 -> ciflow/inductor/158932 2025-10-10T01:23:15.1220756Z * [new tag] ciflow/inductor/159523 -> ciflow/inductor/159523 2025-10-10T01:23:15.1222019Z * [new tag] ciflow/inductor/160266 -> ciflow/inductor/160266 2025-10-10T01:23:15.1223504Z * [new tag] ciflow/inductor/160324 -> ciflow/inductor/160324 2025-10-10T01:23:15.1224764Z * [new tag] ciflow/inductor/160325 -> ciflow/inductor/160325 2025-10-10T01:23:15.1226235Z * [new tag] ciflow/inductor/160326 -> ciflow/inductor/160326 2025-10-10T01:23:15.1227451Z * [new tag] ciflow/inductor/160327 -> ciflow/inductor/160327 2025-10-10T01:23:15.1228887Z * [new tag] ciflow/inductor/160328 -> ciflow/inductor/160328 2025-10-10T01:23:15.1230357Z * [new tag] ciflow/inductor/160329 -> ciflow/inductor/160329 2025-10-10T01:23:15.1232107Z * [new tag] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-10-10T01:23:15.1233365Z * [new tag] ciflow/inductor/160611 -> ciflow/inductor/160611 2025-10-10T01:23:15.1234481Z * [new tag] ciflow/inductor/160843 -> ciflow/inductor/160843 2025-10-10T01:23:15.1235849Z * [new tag] ciflow/inductor/160903 -> ciflow/inductor/160903 2025-10-10T01:23:15.1236992Z * [new tag] ciflow/inductor/161118 -> ciflow/inductor/161118 2025-10-10T01:23:15.1238321Z * [new tag] ciflow/inductor/161158 -> ciflow/inductor/161158 2025-10-10T01:23:15.1239506Z * [new tag] ciflow/inductor/161280 -> ciflow/inductor/161280 2025-10-10T01:23:15.1240987Z * [new tag] ciflow/inductor/161320 -> ciflow/inductor/161320 2025-10-10T01:23:15.1242376Z * [new tag] ciflow/inductor/161485 -> ciflow/inductor/161485 2025-10-10T01:23:15.1243454Z * [new tag] ciflow/inductor/161495 -> ciflow/inductor/161495 2025-10-10T01:23:15.1244803Z * [new tag] ciflow/inductor/161512 -> ciflow/inductor/161512 2025-10-10T01:23:15.1245926Z * [new tag] ciflow/inductor/162031 -> ciflow/inductor/162031 2025-10-10T01:23:15.1247164Z * [new tag] ciflow/inductor/162066 -> ciflow/inductor/162066 2025-10-10T01:23:15.1248437Z * [new tag] ciflow/inductor/162294 -> ciflow/inductor/162294 2025-10-10T01:23:15.1249568Z * [new tag] ciflow/inductor/162340 -> ciflow/inductor/162340 2025-10-10T01:23:15.1250950Z * [new tag] ciflow/inductor/162470 -> ciflow/inductor/162470 2025-10-10T01:23:15.1252314Z * [new tag] ciflow/inductor/162523 -> ciflow/inductor/162523 2025-10-10T01:23:15.1253419Z * [new tag] ciflow/inductor/162542 -> ciflow/inductor/162542 2025-10-10T01:23:15.1254750Z * [new tag] ciflow/inductor/162768 -> ciflow/inductor/162768 2025-10-10T01:23:15.1255881Z * [new tag] ciflow/inductor/162899 -> ciflow/inductor/162899 2025-10-10T01:23:15.1257218Z * [new tag] ciflow/inductor/162900 -> ciflow/inductor/162900 2025-10-10T01:23:15.1258307Z * [new tag] ciflow/inductor/162901 -> ciflow/inductor/162901 2025-10-10T01:23:15.1259650Z * [new tag] ciflow/inductor/162903 -> ciflow/inductor/162903 2025-10-10T01:23:15.1260745Z * [new tag] ciflow/inductor/162905 -> ciflow/inductor/162905 2025-10-10T01:23:15.1262508Z * [new tag] ciflow/inductor/162954 -> ciflow/inductor/162954 2025-10-10T01:23:15.1263632Z * [new tag] ciflow/inductor/162990 -> ciflow/inductor/162990 2025-10-10T01:23:15.1264979Z * [new tag] ciflow/inductor/163027 -> ciflow/inductor/163027 2025-10-10T01:23:15.1266076Z * [new tag] ciflow/inductor/163028 -> ciflow/inductor/163028 2025-10-10T01:23:15.1267437Z * [new tag] ciflow/inductor/163053 -> ciflow/inductor/163053 2025-10-10T01:23:15.1268502Z * [new tag] ciflow/inductor/163185 -> ciflow/inductor/163185 2025-10-10T01:23:15.1269862Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-10-10T01:23:15.1271010Z * [new tag] ciflow/inductor/163490 -> ciflow/inductor/163490 2025-10-10T01:23:15.1272344Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-10-10T01:23:15.1273788Z * [new tag] ciflow/inductor/163517 -> ciflow/inductor/163517 2025-10-10T01:23:15.1274882Z * [new tag] ciflow/inductor/163527 -> ciflow/inductor/163527 2025-10-10T01:23:15.1276241Z * [new tag] ciflow/inductor/163533 -> ciflow/inductor/163533 2025-10-10T01:23:15.1277339Z * [new tag] ciflow/inductor/163602 -> ciflow/inductor/163602 2025-10-10T01:23:15.1278822Z * [new tag] ciflow/inductor/163617 -> ciflow/inductor/163617 2025-10-10T01:23:15.1279891Z * [new tag] ciflow/inductor/163667 -> ciflow/inductor/163667 2025-10-10T01:23:15.1281306Z * [new tag] ciflow/inductor/163671 -> ciflow/inductor/163671 2025-10-10T01:23:15.1282354Z * [new tag] ciflow/inductor/163767 -> ciflow/inductor/163767 2025-10-10T01:23:15.1283705Z * [new tag] ciflow/inductor/163772 -> ciflow/inductor/163772 2025-10-10T01:23:15.1284829Z * [new tag] ciflow/inductor/163806 -> ciflow/inductor/163806 2025-10-10T01:23:15.1286205Z * [new tag] ciflow/inductor/163936 -> ciflow/inductor/163936 2025-10-10T01:23:15.1287541Z * [new tag] ciflow/inductor/163976 -> ciflow/inductor/163976 2025-10-10T01:23:15.1288945Z * [new tag] ciflow/inductor/164039 -> ciflow/inductor/164039 2025-10-10T01:23:15.1290299Z * [new tag] ciflow/inductor/164040 -> ciflow/inductor/164040 2025-10-10T01:23:15.1291651Z * [new tag] ciflow/inductor/164130 -> ciflow/inductor/164130 2025-10-10T01:23:15.1292716Z * [new tag] ciflow/inductor/164144 -> ciflow/inductor/164144 2025-10-10T01:23:15.1293911Z * [new tag] ciflow/inductor/164202 -> ciflow/inductor/164202 2025-10-10T01:23:15.1295179Z * [new tag] ciflow/inductor/164212 -> ciflow/inductor/164212 2025-10-10T01:23:15.1296465Z * [new tag] ciflow/inductor/164273 -> ciflow/inductor/164273 2025-10-10T01:23:15.1297946Z * [new tag] ciflow/inductor/164277 -> ciflow/inductor/164277 2025-10-10T01:23:15.1299011Z * [new tag] ciflow/inductor/164291 -> ciflow/inductor/164291 2025-10-10T01:23:15.1300279Z * [new tag] ciflow/inductor/164296 -> ciflow/inductor/164296 2025-10-10T01:23:15.1301548Z * [new tag] ciflow/inductor/164304 -> ciflow/inductor/164304 2025-10-10T01:23:15.1302634Z * [new tag] ciflow/inductor/164318 -> ciflow/inductor/164318 2025-10-10T01:23:15.1303978Z * [new tag] ciflow/inductor/164321 -> ciflow/inductor/164321 2025-10-10T01:23:15.1305098Z * [new tag] ciflow/inductor/164324 -> ciflow/inductor/164324 2025-10-10T01:23:15.1306451Z * [new tag] ciflow/inductor/164341 -> ciflow/inductor/164341 2025-10-10T01:23:15.1307519Z * [new tag] ciflow/inductor/164343 -> ciflow/inductor/164343 2025-10-10T01:23:15.1308879Z * [new tag] ciflow/inductor/164344 -> ciflow/inductor/164344 2025-10-10T01:23:15.1310172Z * [new tag] ciflow/inductor/164359 -> ciflow/inductor/164359 2025-10-10T01:23:15.1311252Z * [new tag] ciflow/inductor/164373 -> ciflow/inductor/164373 2025-10-10T01:23:15.1312656Z * [new tag] ciflow/inductor/164379 -> ciflow/inductor/164379 2025-10-10T01:23:15.1313778Z * [new tag] ciflow/inductor/164384 -> ciflow/inductor/164384 2025-10-10T01:23:15.1315125Z * [new tag] ciflow/inductor/164404 -> ciflow/inductor/164404 2025-10-10T01:23:15.1316268Z * [new tag] ciflow/inductor/164405 -> ciflow/inductor/164405 2025-10-10T01:23:15.1317530Z * [new tag] ciflow/inductor/164414 -> ciflow/inductor/164414 2025-10-10T01:23:15.1318655Z * [new tag] ciflow/inductor/164422 -> ciflow/inductor/164422 2025-10-10T01:23:15.1320134Z * [new tag] ciflow/inductor/164433 -> ciflow/inductor/164433 2025-10-10T01:23:15.1321234Z * [new tag] ciflow/inductor/164474 -> ciflow/inductor/164474 2025-10-10T01:23:15.1322582Z * [new tag] ciflow/inductor/164488 -> ciflow/inductor/164488 2025-10-10T01:23:15.1323696Z * [new tag] ciflow/inductor/164492 -> ciflow/inductor/164492 2025-10-10T01:23:15.1325237Z * [new tag] ciflow/inductor/164497 -> ciflow/inductor/164497 2025-10-10T01:23:15.1326219Z * [new tag] ciflow/inductor/164498 -> ciflow/inductor/164498 2025-10-10T01:23:15.1327582Z * [new tag] ciflow/inductor/164500 -> ciflow/inductor/164500 2025-10-10T01:23:15.1328647Z * [new tag] ciflow/inductor/164507 -> ciflow/inductor/164507 2025-10-10T01:23:15.1330026Z * [new tag] ciflow/inductor/164519 -> ciflow/inductor/164519 2025-10-10T01:23:15.1331449Z * [new tag] ciflow/inductor/164521 -> ciflow/inductor/164521 2025-10-10T01:23:15.1332540Z * [new tag] ciflow/inductor/164522 -> ciflow/inductor/164522 2025-10-10T01:23:15.1333884Z * [new tag] ciflow/inductor/164523 -> ciflow/inductor/164523 2025-10-10T01:23:15.1335018Z * [new tag] ciflow/inductor/164524 -> ciflow/inductor/164524 2025-10-10T01:23:15.1336356Z * [new tag] ciflow/inductor/164525 -> ciflow/inductor/164525 2025-10-10T01:23:15.1337436Z * [new tag] ciflow/inductor/164526 -> ciflow/inductor/164526 2025-10-10T01:23:15.1339048Z * [new tag] ciflow/inductor/164527 -> ciflow/inductor/164527 2025-10-10T01:23:15.1340139Z * [new tag] ciflow/inductor/164533 -> ciflow/inductor/164533 2025-10-10T01:23:15.1341581Z * [new tag] ciflow/inductor/164537 -> ciflow/inductor/164537 2025-10-10T01:23:15.1343199Z * [new tag] ciflow/inductor/164548 -> ciflow/inductor/164548 2025-10-10T01:23:15.1344523Z * [new tag] ciflow/inductor/164557 -> ciflow/inductor/164557 2025-10-10T01:23:15.1345635Z * [new tag] ciflow/inductor/164558 -> ciflow/inductor/164558 2025-10-10T01:23:15.1347081Z * [new tag] ciflow/inductor/164560 -> ciflow/inductor/164560 2025-10-10T01:23:15.1348196Z * [new tag] ciflow/inductor/164565 -> ciflow/inductor/164565 2025-10-10T01:23:15.1349446Z * [new tag] ciflow/inductor/164577 -> ciflow/inductor/164577 2025-10-10T01:23:15.1350881Z * [new tag] ciflow/inductor/164609 -> ciflow/inductor/164609 2025-10-10T01:23:15.1351888Z * [new tag] ciflow/inductor/164610 -> ciflow/inductor/164610 2025-10-10T01:23:15.1353266Z * [new tag] ciflow/inductor/164611 -> ciflow/inductor/164611 2025-10-10T01:23:15.1354354Z * [new tag] ciflow/inductor/164612 -> ciflow/inductor/164612 2025-10-10T01:23:15.1355708Z * [new tag] ciflow/inductor/164613 -> ciflow/inductor/164613 2025-10-10T01:23:15.1356808Z * [new tag] ciflow/inductor/164614 -> ciflow/inductor/164614 2025-10-10T01:23:15.1358156Z * [new tag] ciflow/inductor/164623 -> ciflow/inductor/164623 2025-10-10T01:23:15.1359322Z * [new tag] ciflow/inductor/164626 -> ciflow/inductor/164626 2025-10-10T01:23:15.1360706Z * [new tag] ciflow/inductor/164628 -> ciflow/inductor/164628 2025-10-10T01:23:15.1361978Z * [new tag] ciflow/inductor/164631 -> ciflow/inductor/164631 2025-10-10T01:23:15.1363345Z * [new tag] ciflow/inductor/164632 -> ciflow/inductor/164632 2025-10-10T01:23:15.1364471Z * [new tag] ciflow/inductor/164633 -> ciflow/inductor/164633 2025-10-10T01:23:15.1365809Z * [new tag] ciflow/inductor/164640 -> ciflow/inductor/164640 2025-10-10T01:23:15.1367104Z * [new tag] ciflow/inductor/164641 -> ciflow/inductor/164641 2025-10-10T01:23:15.1368463Z * [new tag] ciflow/inductor/164645 -> ciflow/inductor/164645 2025-10-10T01:23:15.1369590Z * [new tag] ciflow/inductor/164648 -> ciflow/inductor/164648 2025-10-10T01:23:15.1371156Z * [new tag] ciflow/inductor/164653 -> ciflow/inductor/164653 2025-10-10T01:23:15.1372265Z * [new tag] ciflow/inductor/164655 -> ciflow/inductor/164655 2025-10-10T01:23:15.1373698Z * [new tag] ciflow/inductor/164657 -> ciflow/inductor/164657 2025-10-10T01:23:15.1374967Z * [new tag] ciflow/inductor/164659 -> ciflow/inductor/164659 2025-10-10T01:23:15.1376304Z * [new tag] ciflow/inductor/164669 -> ciflow/inductor/164669 2025-10-10T01:23:15.1377632Z * [new tag] ciflow/inductor/164690 -> ciflow/inductor/164690 2025-10-10T01:23:15.1378745Z * [new tag] ciflow/inductor/164691 -> ciflow/inductor/164691 2025-10-10T01:23:15.1380256Z * [new tag] ciflow/inductor/164692 -> ciflow/inductor/164692 2025-10-10T01:23:15.1381410Z * [new tag] ciflow/inductor/164711 -> ciflow/inductor/164711 2025-10-10T01:23:15.1382774Z * [new tag] ciflow/inductor/164714 -> ciflow/inductor/164714 2025-10-10T01:23:15.1383866Z * [new tag] ciflow/inductor/164717 -> ciflow/inductor/164717 2025-10-10T01:23:15.1385200Z * [new tag] ciflow/inductor/164718 -> ciflow/inductor/164718 2025-10-10T01:23:15.1386648Z * [new tag] ciflow/inductor/164723 -> ciflow/inductor/164723 2025-10-10T01:23:15.1387751Z * [new tag] ciflow/inductor/164724 -> ciflow/inductor/164724 2025-10-10T01:23:15.1389116Z * [new tag] ciflow/inductor/164734 -> ciflow/inductor/164734 2025-10-10T01:23:15.1390384Z * [new tag] ciflow/inductor/164740 -> ciflow/inductor/164740 2025-10-10T01:23:15.1391787Z * [new tag] ciflow/inductor/164746 -> ciflow/inductor/164746 2025-10-10T01:23:15.1392932Z * [new tag] ciflow/inductor/164747 -> ciflow/inductor/164747 2025-10-10T01:23:15.1394275Z * [new tag] ciflow/inductor/164776 -> ciflow/inductor/164776 2025-10-10T01:23:15.1395415Z * [new tag] ciflow/inductor/164778 -> ciflow/inductor/164778 2025-10-10T01:23:15.1396914Z * [new tag] ciflow/inductor/164780 -> ciflow/inductor/164780 2025-10-10T01:23:15.1399791Z * [new tag] ciflow/inductor/164794 -> ciflow/inductor/164794 2025-10-10T01:23:15.1400887Z * [new tag] ciflow/inductor/164802 -> ciflow/inductor/164802 2025-10-10T01:23:15.1402178Z * [new tag] ciflow/inductor/164806 -> ciflow/inductor/164806 2025-10-10T01:23:15.1403597Z * [new tag] ciflow/inductor/164808 -> ciflow/inductor/164808 2025-10-10T01:23:15.1404766Z * [new tag] ciflow/inductor/164810 -> ciflow/inductor/164810 2025-10-10T01:23:15.1406132Z * [new tag] ciflow/inductor/164811 -> ciflow/inductor/164811 2025-10-10T01:23:15.1407292Z * [new tag] ciflow/inductor/164812 -> ciflow/inductor/164812 2025-10-10T01:23:15.1408572Z * [new tag] ciflow/inductor/164819 -> ciflow/inductor/164819 2025-10-10T01:23:15.1409705Z * [new tag] ciflow/inductor/164820 -> ciflow/inductor/164820 2025-10-10T01:23:15.1411064Z * [new tag] ciflow/inductor/164821 -> ciflow/inductor/164821 2025-10-10T01:23:15.1412231Z * [new tag] ciflow/inductor/164839 -> ciflow/inductor/164839 2025-10-10T01:23:15.1413510Z * [new tag] ciflow/inductor/164842 -> ciflow/inductor/164842 2025-10-10T01:23:15.1414651Z * [new tag] ciflow/inductor/164847 -> ciflow/inductor/164847 2025-10-10T01:23:15.1415999Z * [new tag] ciflow/inductor/164852 -> ciflow/inductor/164852 2025-10-10T01:23:15.1417158Z * [new tag] ciflow/inductor/164863 -> ciflow/inductor/164863 2025-10-10T01:23:15.1418504Z * [new tag] ciflow/inductor/164865 -> ciflow/inductor/164865 2025-10-10T01:23:15.1419786Z * [new tag] ciflow/inductor/164866 -> ciflow/inductor/164866 2025-10-10T01:23:15.1420826Z * [new tag] ciflow/inductor/164867 -> ciflow/inductor/164867 2025-10-10T01:23:15.1422018Z * [new tag] ciflow/inductor/164869 -> ciflow/inductor/164869 2025-10-10T01:23:15.1423365Z * [new tag] ciflow/inductor/164873 -> ciflow/inductor/164873 2025-10-10T01:23:15.1424483Z * [new tag] ciflow/inductor/164889 -> ciflow/inductor/164889 2025-10-10T01:23:15.1426169Z * [new tag] ciflow/inductor/164897 -> ciflow/inductor/164897 2025-10-10T01:23:15.1427307Z * [new tag] ciflow/inductor/164902 -> ciflow/inductor/164902 2025-10-10T01:23:15.1428703Z * [new tag] ciflow/inductor/164903 -> ciflow/inductor/164903 2025-10-10T01:23:15.1429824Z * [new tag] ciflow/inductor/164906 -> ciflow/inductor/164906 2025-10-10T01:23:15.1431201Z * [new tag] ciflow/inductor/164914 -> ciflow/inductor/164914 2025-10-10T01:23:15.1432348Z * [new tag] ciflow/inductor/164919 -> ciflow/inductor/164919 2025-10-10T01:23:15.1433702Z * [new tag] ciflow/inductor/164933 -> ciflow/inductor/164933 2025-10-10T01:23:15.1434925Z * [new tag] ciflow/inductor/164938 -> ciflow/inductor/164938 2025-10-10T01:23:15.1436214Z * [new tag] ciflow/inductor/164948 -> ciflow/inductor/164948 2025-10-10T01:23:15.1437369Z * [new tag] ciflow/inductor/164956 -> ciflow/inductor/164956 2025-10-10T01:23:15.1438726Z * [new tag] ciflow/inductor/164965 -> ciflow/inductor/164965 2025-10-10T01:23:15.1439916Z * [new tag] ciflow/inductor/164978 -> ciflow/inductor/164978 2025-10-10T01:23:15.1441348Z * [new tag] ciflow/inductor/164979 -> ciflow/inductor/164979 2025-10-10T01:23:15.1442447Z * [new tag] ciflow/inductor/164980 -> ciflow/inductor/164980 2025-10-10T01:23:15.1443907Z * [new tag] ciflow/inductor/164984 -> ciflow/inductor/164984 2025-10-10T01:23:15.1445180Z * [new tag] ciflow/inductor/164989 -> ciflow/inductor/164989 2025-10-10T01:23:15.1446321Z * [new tag] ciflow/inductor/164991 -> ciflow/inductor/164991 2025-10-10T01:23:15.1447681Z * [new tag] ciflow/inductor/164992 -> ciflow/inductor/164992 2025-10-10T01:23:15.1449034Z * [new tag] ciflow/inductor/164994 -> ciflow/inductor/164994 2025-10-10T01:23:15.1450187Z * [new tag] ciflow/inductor/164999 -> ciflow/inductor/164999 2025-10-10T01:23:15.1451535Z * [new tag] ciflow/inductor/165001 -> ciflow/inductor/165001 2025-10-10T01:23:15.1452703Z * [new tag] ciflow/inductor/165005 -> ciflow/inductor/165005 2025-10-10T01:23:15.1454053Z * [new tag] ciflow/inductor/165006 -> ciflow/inductor/165006 2025-10-10T01:23:15.1455173Z * [new tag] ciflow/inductor/165012 -> ciflow/inductor/165012 2025-10-10T01:23:15.1456446Z * [new tag] ciflow/inductor/165017 -> ciflow/inductor/165017 2025-10-10T01:23:15.1457883Z * [new tag] ciflow/inductor/165018 -> ciflow/inductor/165018 2025-10-10T01:23:15.1458994Z * [new tag] ciflow/inductor/165024 -> ciflow/inductor/165024 2025-10-10T01:23:15.1460420Z * [new tag] ciflow/inductor/165029 -> ciflow/inductor/165029 2025-10-10T01:23:15.1461797Z * [new tag] ciflow/inductor/165030 -> ciflow/inductor/165030 2025-10-10T01:23:15.1463146Z * [new tag] ciflow/inductor/165031 -> ciflow/inductor/165031 2025-10-10T01:23:15.1464511Z * [new tag] ciflow/inductor/165033 -> ciflow/inductor/165033 2025-10-10T01:23:15.1466048Z * [new tag] ciflow/inductor/165036 -> ciflow/inductor/165036 2025-10-10T01:23:15.1467329Z * [new tag] ciflow/inductor/165037 -> ciflow/inductor/165037 2025-10-10T01:23:15.1468437Z * [new tag] ciflow/inductor/165039 -> ciflow/inductor/165039 2025-10-10T01:23:15.1469893Z * [new tag] ciflow/inductor/165047 -> ciflow/inductor/165047 2025-10-10T01:23:15.1470983Z * [new tag] ciflow/inductor/165059 -> ciflow/inductor/165059 2025-10-10T01:23:15.1472342Z * [new tag] ciflow/inductor/165063 -> ciflow/inductor/165063 2025-10-10T01:23:15.1473437Z * [new tag] ciflow/inductor/165064 -> ciflow/inductor/165064 2025-10-10T01:23:15.1474776Z * [new tag] ciflow/inductor/165066 -> ciflow/inductor/165066 2025-10-10T01:23:15.1475901Z * [new tag] ciflow/inductor/165074 -> ciflow/inductor/165074 2025-10-10T01:23:15.1477292Z * [new tag] ciflow/inductor/165076 -> ciflow/inductor/165076 2025-10-10T01:23:15.1478438Z * [new tag] ciflow/inductor/165091 -> ciflow/inductor/165091 2025-10-10T01:23:15.1479855Z * [new tag] ciflow/inductor/165092 -> ciflow/inductor/165092 2025-10-10T01:23:15.1480905Z * [new tag] ciflow/inductor/165106 -> ciflow/inductor/165106 2025-10-10T01:23:15.1482287Z * [new tag] ciflow/inductor/165107 -> ciflow/inductor/165107 2025-10-10T01:23:15.1483663Z * [new tag] ciflow/inductor/165112 -> ciflow/inductor/165112 2025-10-10T01:23:15.1484813Z * [new tag] ciflow/inductor/165113 -> ciflow/inductor/165113 2025-10-10T01:23:15.1486248Z * [new tag] ciflow/inductor/165114 -> ciflow/inductor/165114 2025-10-10T01:23:15.1487755Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-10-10T01:23:15.1489153Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-10-10T01:23:15.1490561Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-10-10T01:23:15.1492243Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-10-10T01:23:15.1493570Z * [new tag] ciflow/linux-aarch64/163952 -> ciflow/linux-aarch64/163952 2025-10-10T01:23:15.1494699Z * [new tag] ciflow/linux-aarch64/164965 -> ciflow/linux-aarch64/164965 2025-10-10T01:23:15.1496942Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-10-10T01:23:15.1497932Z * [new tag] ciflow/mps/157554 -> ciflow/mps/157554 2025-10-10T01:23:15.1499216Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-10-10T01:23:15.1500156Z * [new tag] ciflow/mps/162340 -> ciflow/mps/162340 2025-10-10T01:23:15.1501625Z * [new tag] ciflow/mps/164416 -> ciflow/mps/164416 2025-10-10T01:23:15.1503050Z * [new tag] ciflow/mps/164571 -> ciflow/mps/164571 2025-10-10T01:23:15.1504056Z * [new tag] ciflow/mps/164965 -> ciflow/mps/164965 2025-10-10T01:23:15.1505616Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-10-10T01:23:15.1506621Z * [new tag] ciflow/nightly/164747 -> ciflow/nightly/164747 2025-10-10T01:23:15.1507836Z * [new tag] ciflow/nightly/164901 -> ciflow/nightly/164901 2025-10-10T01:23:15.1509328Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-10-10T01:23:15.1510341Z * [new tag] ciflow/op-benchmark/163767 -> ciflow/op-benchmark/163767 2025-10-10T01:23:15.1511500Z * [new tag] ciflow/op-benchmark/164583 -> ciflow/op-benchmark/164583 2025-10-10T01:23:15.1512596Z * [new tag] ciflow/op-benchmark/164747 -> ciflow/op-benchmark/164747 2025-10-10T01:23:15.1514774Z * [new tag] ciflow/periodic-rocm-mi300/162478 -> ciflow/periodic-rocm-mi300/162478 2025-10-10T01:23:15.1515610Z * [new tag] ciflow/periodic-rocm-mi300/163767 -> ciflow/periodic-rocm-mi300/163767 2025-10-10T01:23:15.1516840Z * [new tag] ciflow/periodic-rocm-mi300/164618 -> ciflow/periodic-rocm-mi300/164618 2025-10-10T01:23:15.1518018Z * [new tag] ciflow/periodic-rocm-mi300/164747 -> ciflow/periodic-rocm-mi300/164747 2025-10-10T01:23:15.1519302Z * [new tag] ciflow/periodic-rocm-mi300/165011 -> ciflow/periodic-rocm-mi300/165011 2025-10-10T01:23:15.1520518Z * [new tag] ciflow/periodic-rocm-mi300/165080 -> ciflow/periodic-rocm-mi300/165080 2025-10-10T01:23:15.1521738Z * [new tag] ciflow/periodic-rocm-mi300/165115 -> ciflow/periodic-rocm-mi300/165115 2025-10-10T01:23:15.1523441Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-10-10T01:23:15.1524905Z * [new tag] ciflow/periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 -> ciflow/periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 2025-10-10T01:23:15.1526266Z * [new tag] ciflow/periodic/0ea59c3c55dab37a6edefcc7002bb1428afd6456 -> ciflow/periodic/0ea59c3c55dab37a6edefcc7002bb1428afd6456 2025-10-10T01:23:15.1527175Z * [new tag] ciflow/periodic/156491 -> ciflow/periodic/156491 2025-10-10T01:23:15.1528501Z * [new tag] ciflow/periodic/162990 -> ciflow/periodic/162990 2025-10-10T01:23:15.1529459Z * [new tag] ciflow/periodic/163667 -> ciflow/periodic/163667 2025-10-10T01:23:15.1530594Z * [new tag] ciflow/periodic/163767 -> ciflow/periodic/163767 2025-10-10T01:23:15.1532195Z * [new tag] ciflow/periodic/164747 -> ciflow/periodic/164747 2025-10-10T01:23:15.1533684Z * [new tag] ciflow/periodic/164769 -> ciflow/periodic/164769 2025-10-10T01:23:15.1534767Z * [new tag] ciflow/periodic/165011 -> ciflow/periodic/165011 2025-10-10T01:23:15.1536724Z * [new tag] ciflow/periodic/2a6cdba6e5f74c2294fecc2d1344537522efbaab -> ciflow/periodic/2a6cdba6e5f74c2294fecc2d1344537522efbaab 2025-10-10T01:23:15.1537635Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-10-10T01:23:15.1539121Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-10-10T01:23:15.1540462Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-10-10T01:23:15.1541950Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-10-10T01:23:15.1543651Z * [new tag] ciflow/periodic/4bcc05777e780e834d44a2d06dd5321daec316f0 -> ciflow/periodic/4bcc05777e780e834d44a2d06dd5321daec316f0 2025-10-10T01:23:15.1544793Z * [new tag] ciflow/periodic/73adac05d13babb75410c3e033fdce57aa16881a -> ciflow/periodic/73adac05d13babb75410c3e033fdce57aa16881a 2025-10-10T01:23:15.1546589Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-10-10T01:23:15.1548319Z * [new tag] ciflow/periodic/ac08556f674259ff5b117964e300124e8a92d45b -> ciflow/periodic/ac08556f674259ff5b117964e300124e8a92d45b 2025-10-10T01:23:15.1549684Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-10-10T01:23:15.1551083Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-10-10T01:23:15.1552371Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-10-10T01:23:15.1553876Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-10-10T01:23:15.1555436Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-10-10T01:23:15.1556896Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-10-10T01:23:15.1558554Z * [new tag] ciflow/quantization-periodic/163767 -> ciflow/quantization-periodic/163767 2025-10-10T01:23:15.1559516Z * [new tag] ciflow/quantization-periodic/164747 -> ciflow/quantization-periodic/164747 2025-10-10T01:23:15.1561084Z * [new tag] ciflow/riscv64/163767 -> ciflow/riscv64/163767 2025-10-10T01:23:15.1562041Z * [new tag] ciflow/riscv64/164747 -> ciflow/riscv64/164747 2025-10-10T01:23:15.1563616Z * [new tag] ciflow/rocm-mi300/161280 -> ciflow/rocm-mi300/161280 2025-10-10T01:23:15.1564608Z * [new tag] ciflow/rocm-mi300/162478 -> ciflow/rocm-mi300/162478 2025-10-10T01:23:15.1565735Z * [new tag] ciflow/rocm-mi300/163767 -> ciflow/rocm-mi300/163767 2025-10-10T01:23:15.1566877Z * [new tag] ciflow/rocm-mi300/163955 -> ciflow/rocm-mi300/163955 2025-10-10T01:23:15.1567986Z * [new tag] ciflow/rocm-mi300/164618 -> ciflow/rocm-mi300/164618 2025-10-10T01:23:15.1569101Z * [new tag] ciflow/rocm-mi300/164747 -> ciflow/rocm-mi300/164747 2025-10-10T01:23:15.1570543Z * [new tag] ciflow/rocm-mi300/164927 -> ciflow/rocm-mi300/164927 2025-10-10T01:23:15.1571620Z * [new tag] ciflow/rocm-mi300/164930 -> ciflow/rocm-mi300/164930 2025-10-10T01:23:15.1573027Z * [new tag] ciflow/rocm-mi300/165026 -> ciflow/rocm-mi300/165026 2025-10-10T01:23:15.1574067Z * [new tag] ciflow/rocm-mi300/165080 -> ciflow/rocm-mi300/165080 2025-10-10T01:23:15.1575217Z * [new tag] ciflow/rocm-mi300/165115 -> ciflow/rocm-mi300/165115 2025-10-10T01:23:15.1576721Z * [new tag] ciflow/rocm-mi355/163767 -> ciflow/rocm-mi355/163767 2025-10-10T01:23:15.1577680Z * [new tag] ciflow/rocm-mi355/164747 -> ciflow/rocm-mi355/164747 2025-10-10T01:23:15.1579234Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-10-10T01:23:15.1580284Z * [new tag] ciflow/rocm/151845 -> ciflow/rocm/151845 2025-10-10T01:23:15.1581393Z * [new tag] ciflow/rocm/156592 -> ciflow/rocm/156592 2025-10-10T01:23:15.1582519Z * [new tag] ciflow/rocm/161280 -> ciflow/rocm/161280 2025-10-10T01:23:15.1583638Z * [new tag] ciflow/rocm/163767 -> ciflow/rocm/163767 2025-10-10T01:23:15.1584969Z * [new tag] ciflow/rocm/163955 -> ciflow/rocm/163955 2025-10-10T01:23:15.1586062Z * [new tag] ciflow/rocm/163965 -> ciflow/rocm/163965 2025-10-10T01:23:15.1587433Z * [new tag] ciflow/rocm/164656 -> ciflow/rocm/164656 2025-10-10T01:23:15.1588469Z * [new tag] ciflow/rocm/164747 -> ciflow/rocm/164747 2025-10-10T01:23:15.1589715Z * [new tag] ciflow/rocm/164769 -> ciflow/rocm/164769 2025-10-10T01:23:15.1590829Z * [new tag] ciflow/rocm/164927 -> ciflow/rocm/164927 2025-10-10T01:23:15.1592395Z * [new tag] ciflow/rocm/164930 -> ciflow/rocm/164930 2025-10-10T01:23:15.1593912Z * [new tag] ciflow/rocm/165026 -> ciflow/rocm/165026 2025-10-10T01:23:15.1595428Z * [new tag] ciflow/s390/164747 -> ciflow/s390/164747 2025-10-10T01:23:15.1596991Z * [new tag] ciflow/s390/164917 -> ciflow/s390/164917 2025-10-10T01:23:15.1599621Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-10-10T01:23:15.1601458Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-10-10T01:23:15.1603347Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-10-10T01:23:15.1604280Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-10-10T01:23:15.1605821Z * [new tag] ciflow/slow/163767 -> ciflow/slow/163767 2025-10-10T01:23:15.1606687Z * [new tag] ciflow/slow/164747 -> ciflow/slow/164747 2025-10-10T01:23:15.1608081Z * [new tag] ciflow/slow/164769 -> ciflow/slow/164769 2025-10-10T01:23:15.1609419Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-10-10T01:23:15.1611255Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-10-10T01:23:15.1613197Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-10-10T01:23:15.1614935Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-10-10T01:23:15.1616329Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-10-10T01:23:15.1617791Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-10-10T01:23:15.1619207Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-10-10T01:23:15.1620667Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-10-10T01:23:15.1622689Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-10-10T01:23:15.1623929Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-10-10T01:23:15.1625284Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-10-10T01:23:15.1626783Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-10-10T01:23:15.1628189Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-10-10T01:23:15.1629645Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-10-10T01:23:15.1631406Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-10-10T01:23:15.1640868Z * [new tag] ciflow/torchbench/164747 -> ciflow/torchbench/164747 2025-10-10T01:23:15.1641114Z * [new tag] ciflow/trunk/113258 -> ciflow/trunk/113258 2025-10-10T01:23:15.1641291Z * [new tag] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-10-10T01:23:15.1641419Z * [new tag] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-10-10T01:23:15.1641548Z * [new tag] ciflow/trunk/148328 -> ciflow/trunk/148328 2025-10-10T01:23:15.1641664Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-10-10T01:23:15.1641777Z * [new tag] ciflow/trunk/149003 -> ciflow/trunk/149003 2025-10-10T01:23:15.1641906Z * [new tag] ciflow/trunk/149536 -> ciflow/trunk/149536 2025-10-10T01:23:15.1642649Z * [new tag] ciflow/trunk/151845 -> ciflow/trunk/151845 2025-10-10T01:23:15.1644453Z * [new tag] ciflow/trunk/152624 -> ciflow/trunk/152624 2025-10-10T01:23:15.1645693Z * [new tag] ciflow/trunk/154279 -> ciflow/trunk/154279 2025-10-10T01:23:15.1646897Z * [new tag] ciflow/trunk/154983 -> ciflow/trunk/154983 2025-10-10T01:23:15.1648515Z * [new tag] ciflow/trunk/156418 -> ciflow/trunk/156418 2025-10-10T01:23:15.1650100Z * [new tag] ciflow/trunk/156592 -> ciflow/trunk/156592 2025-10-10T01:23:15.1651711Z * [new tag] ciflow/trunk/157432 -> ciflow/trunk/157432 2025-10-10T01:23:15.1652974Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-10-10T01:23:15.1654224Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-10-10T01:23:15.1655551Z * [new tag] ciflow/trunk/158932 -> ciflow/trunk/158932 2025-10-10T01:23:15.1656958Z * [new tag] ciflow/trunk/159104 -> ciflow/trunk/159104 2025-10-10T01:23:15.1658293Z * [new tag] ciflow/trunk/159936 -> ciflow/trunk/159936 2025-10-10T01:23:15.1659570Z * [new tag] ciflow/trunk/160266 -> ciflow/trunk/160266 2025-10-10T01:23:15.1660804Z * [new tag] ciflow/trunk/160328 -> ciflow/trunk/160328 2025-10-10T01:23:15.1662064Z * [new tag] ciflow/trunk/160329 -> ciflow/trunk/160329 2025-10-10T01:23:15.1663315Z * [new tag] ciflow/trunk/160539 -> ciflow/trunk/160539 2025-10-10T01:23:15.1664851Z * [new tag] ciflow/trunk/160610 -> ciflow/trunk/160610 2025-10-10T01:23:15.1666090Z * [new tag] ciflow/trunk/160843 -> ciflow/trunk/160843 2025-10-10T01:23:15.1667495Z * [new tag] ciflow/trunk/161035 -> ciflow/trunk/161035 2025-10-10T01:23:15.1668742Z * [new tag] ciflow/trunk/161320 -> ciflow/trunk/161320 2025-10-10T01:23:15.1670010Z * [new tag] ciflow/trunk/162031 -> ciflow/trunk/162031 2025-10-10T01:23:15.1671260Z * [new tag] ciflow/trunk/162066 -> ciflow/trunk/162066 2025-10-10T01:23:15.1672704Z * [new tag] ciflow/trunk/162203 -> ciflow/trunk/162203 2025-10-10T01:23:15.1674002Z * [new tag] ciflow/trunk/162340 -> ciflow/trunk/162340 2025-10-10T01:23:15.1675266Z * [new tag] ciflow/trunk/162542 -> ciflow/trunk/162542 2025-10-10T01:23:15.1676534Z * [new tag] ciflow/trunk/162899 -> ciflow/trunk/162899 2025-10-10T01:23:15.1677984Z * [new tag] ciflow/trunk/163034 -> ciflow/trunk/163034 2025-10-10T01:23:15.1679567Z * [new tag] ciflow/trunk/163332 -> ciflow/trunk/163332 2025-10-10T01:23:15.1680957Z * [new tag] ciflow/trunk/163446 -> ciflow/trunk/163446 2025-10-10T01:23:15.1682197Z * [new tag] ciflow/trunk/163490 -> ciflow/trunk/163490 2025-10-10T01:23:15.1683444Z * [new tag] ciflow/trunk/163527 -> ciflow/trunk/163527 2025-10-10T01:23:15.1684689Z * [new tag] ciflow/trunk/163533 -> ciflow/trunk/163533 2025-10-10T01:23:15.1685976Z * [new tag] ciflow/trunk/163671 -> ciflow/trunk/163671 2025-10-10T01:23:15.1687234Z * [new tag] ciflow/trunk/163767 -> ciflow/trunk/163767 2025-10-10T01:23:15.1688504Z * [new tag] ciflow/trunk/163846 -> ciflow/trunk/163846 2025-10-10T01:23:15.1689756Z * [new tag] ciflow/trunk/163899 -> ciflow/trunk/163899 2025-10-10T01:23:15.1691053Z * [new tag] ciflow/trunk/163955 -> ciflow/trunk/163955 2025-10-10T01:23:15.1692343Z * [new tag] ciflow/trunk/163976 -> ciflow/trunk/163976 2025-10-10T01:23:15.1693609Z * [new tag] ciflow/trunk/164040 -> ciflow/trunk/164040 2025-10-10T01:23:15.1694848Z * [new tag] ciflow/trunk/164130 -> ciflow/trunk/164130 2025-10-10T01:23:15.1696269Z * [new tag] ciflow/trunk/164144 -> ciflow/trunk/164144 2025-10-10T01:23:15.1697625Z * [new tag] ciflow/trunk/164202 -> ciflow/trunk/164202 2025-10-10T01:23:15.1698874Z * [new tag] ciflow/trunk/164318 -> ciflow/trunk/164318 2025-10-10T01:23:15.1700154Z * [new tag] ciflow/trunk/164414 -> ciflow/trunk/164414 2025-10-10T01:23:15.1701385Z * [new tag] ciflow/trunk/164416 -> ciflow/trunk/164416 2025-10-10T01:23:15.1702640Z * [new tag] ciflow/trunk/164437 -> ciflow/trunk/164437 2025-10-10T01:23:15.1703903Z * [new tag] ciflow/trunk/164467 -> ciflow/trunk/164467 2025-10-10T01:23:15.1705180Z * [new tag] ciflow/trunk/164500 -> ciflow/trunk/164500 2025-10-10T01:23:15.1706577Z * [new tag] ciflow/trunk/164510 -> ciflow/trunk/164510 2025-10-10T01:23:15.1707699Z * [new tag] ciflow/trunk/164519 -> ciflow/trunk/164519 2025-10-10T01:23:15.1708933Z * [new tag] ciflow/trunk/164542 -> ciflow/trunk/164542 2025-10-10T01:23:15.1710198Z * [new tag] ciflow/trunk/164560 -> ciflow/trunk/164560 2025-10-10T01:23:15.1711584Z * [new tag] ciflow/trunk/164566 -> ciflow/trunk/164566 2025-10-10T01:23:15.1712821Z * [new tag] ciflow/trunk/164623 -> ciflow/trunk/164623 2025-10-10T01:23:15.1714075Z * [new tag] ciflow/trunk/164628 -> ciflow/trunk/164628 2025-10-10T01:23:15.1715364Z * [new tag] ciflow/trunk/164641 -> ciflow/trunk/164641 2025-10-10T01:23:15.1716798Z * [new tag] ciflow/trunk/164643 -> ciflow/trunk/164643 2025-10-10T01:23:15.1718057Z * [new tag] ciflow/trunk/164645 -> ciflow/trunk/164645 2025-10-10T01:23:15.1719366Z * [new tag] ciflow/trunk/164653 -> ciflow/trunk/164653 2025-10-10T01:23:15.1720778Z * [new tag] ciflow/trunk/164655 -> ciflow/trunk/164655 2025-10-10T01:23:15.1722021Z * [new tag] ciflow/trunk/164691 -> ciflow/trunk/164691 2025-10-10T01:23:15.1723216Z * [new tag] ciflow/trunk/164692 -> ciflow/trunk/164692 2025-10-10T01:23:15.1724457Z * [new tag] ciflow/trunk/164705 -> ciflow/trunk/164705 2025-10-10T01:23:15.1725739Z * [new tag] ciflow/trunk/164746 -> ciflow/trunk/164746 2025-10-10T01:23:15.1726950Z * [new tag] ciflow/trunk/164747 -> ciflow/trunk/164747 2025-10-10T01:23:15.1728774Z * [new tag] ciflow/trunk/164790 -> ciflow/trunk/164790 2025-10-10T01:23:15.1730059Z * [new tag] ciflow/trunk/164808 -> ciflow/trunk/164808 2025-10-10T01:23:15.1731404Z * [new tag] ciflow/trunk/164812 -> ciflow/trunk/164812 2025-10-10T01:23:15.1732822Z * [new tag] ciflow/trunk/164836 -> ciflow/trunk/164836 2025-10-10T01:23:15.1734088Z * [new tag] ciflow/trunk/164842 -> ciflow/trunk/164842 2025-10-10T01:23:15.1735348Z * [new tag] ciflow/trunk/164882 -> ciflow/trunk/164882 2025-10-10T01:23:15.1736646Z * [new tag] ciflow/trunk/164889 -> ciflow/trunk/164889 2025-10-10T01:23:15.1737855Z * [new tag] ciflow/trunk/164894 -> ciflow/trunk/164894 2025-10-10T01:23:15.1739135Z * [new tag] ciflow/trunk/164930 -> ciflow/trunk/164930 2025-10-10T01:23:15.1740395Z * [new tag] ciflow/trunk/164953 -> ciflow/trunk/164953 2025-10-10T01:23:15.1741683Z * [new tag] ciflow/trunk/164976 -> ciflow/trunk/164976 2025-10-10T01:23:15.1742929Z * [new tag] ciflow/trunk/164999 -> ciflow/trunk/164999 2025-10-10T01:23:15.1744186Z * [new tag] ciflow/trunk/165000 -> ciflow/trunk/165000 2025-10-10T01:23:15.1745475Z * [new tag] ciflow/trunk/165017 -> ciflow/trunk/165017 2025-10-10T01:23:15.1746757Z * [new tag] ciflow/trunk/165018 -> ciflow/trunk/165018 2025-10-10T01:23:15.1747981Z * [new tag] ciflow/trunk/165024 -> ciflow/trunk/165024 2025-10-10T01:23:15.1749242Z * [new tag] ciflow/trunk/165031 -> ciflow/trunk/165031 2025-10-10T01:23:15.1750505Z * [new tag] ciflow/trunk/165033 -> ciflow/trunk/165033 2025-10-10T01:23:15.1751818Z * [new tag] ciflow/trunk/165047 -> ciflow/trunk/165047 2025-10-10T01:23:15.1753060Z * [new tag] ciflow/trunk/165057 -> ciflow/trunk/165057 2025-10-10T01:23:15.1754410Z * [new tag] ciflow/trunk/165060 -> ciflow/trunk/165060 2025-10-10T01:23:15.1755569Z * [new tag] ciflow/trunk/165065 -> ciflow/trunk/165065 2025-10-10T01:23:15.1756847Z * [new tag] ciflow/trunk/165066 -> ciflow/trunk/165066 2025-10-10T01:23:15.1758283Z * [new tag] ciflow/trunk/165090 -> ciflow/trunk/165090 2025-10-10T01:23:15.1760006Z * [new tag] ciflow/trunk/165094 -> ciflow/trunk/165094 2025-10-10T01:23:15.1761266Z * [new tag] ciflow/trunk/165113 -> ciflow/trunk/165113 2025-10-10T01:23:15.1763066Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-10-10T01:23:15.1764490Z * [new tag] ciflow/vllm/164628 -> ciflow/vllm/164628 2025-10-10T01:23:15.1765956Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-10-10T01:23:15.1767413Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-10-10T01:23:15.1768659Z * [new tag] ciflow/xpu/161485 -> ciflow/xpu/161485 2025-10-10T01:23:15.1769955Z * [new tag] ciflow/xpu/162454 -> ciflow/xpu/162454 2025-10-10T01:23:15.1771141Z * [new tag] ciflow/xpu/163332 -> ciflow/xpu/163332 2025-10-10T01:23:15.1772324Z * [new tag] cslpull75 -> cslpull75 2025-10-10T01:23:15.1773575Z * [new tag] cslpull76 -> cslpull76 2025-10-10T01:23:15.1774788Z * [new tag] cslpull77 -> cslpull77 2025-10-10T01:23:15.1776035Z * [new tag] cslpull78 -> cslpull78 2025-10-10T01:23:15.1777580Z * [new tag] cslpull79 -> cslpull79 2025-10-10T01:23:15.1779216Z * [new tag] cslpull80 -> cslpull80 2025-10-10T01:23:15.1780601Z * [new tag] cslpull81 -> cslpull81 2025-10-10T01:23:15.1781972Z * [new tag] cslpull82 -> cslpull82 2025-10-10T01:23:15.1783323Z * [new tag] cslpull83 -> cslpull83 2025-10-10T01:23:15.1784708Z * [new tag] cslpull84 -> cslpull84 2025-10-10T01:23:15.1786015Z * [new tag] cslpull85 -> cslpull85 2025-10-10T01:23:15.1787394Z * [new tag] cslpull86 -> cslpull86 2025-10-10T01:23:15.1788724Z * [new tag] cslpull87 -> cslpull87 2025-10-10T01:23:15.1790120Z * [new tag] cslpull88 -> cslpull88 2025-10-10T01:23:15.1791426Z * [new tag] cslpull89 -> cslpull89 2025-10-10T01:23:15.1792608Z * [new tag] cslpull90 -> cslpull90 2025-10-10T01:23:15.1794243Z * [new tag] cslpull91 -> cslpull91 2025-10-10T01:23:15.1795550Z * [new tag] cslpull92 -> cslpull92 2025-10-10T01:23:15.1797123Z * [new tag] flight_5 -> flight_5 2025-10-10T01:23:15.1798487Z * [new tag] flight_5.1 -> flight_5.1 2025-10-10T01:23:15.1799977Z * [new tag] flight_5.2 -> flight_5.2 2025-10-10T01:23:15.1801337Z * [new tag] flight_5.3 -> flight_5.3 2025-10-10T01:23:15.1802621Z * [new tag] forpull1 -> forpull1 2025-10-10T01:23:15.1804212Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-10-10T01:23:15.1805501Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-10-10T01:23:15.1806806Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-10-10T01:23:15.1808192Z * [new tag] nightly-binary -> nightly-binary 2025-10-10T01:23:15.1809515Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-10-10T01:23:15.1810906Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-10-10T01:23:15.1812603Z * [new tag] trunk/001e1d263746ae9d121d9c8cf55bc87f777d9dba -> trunk/001e1d263746ae9d121d9c8cf55bc87f777d9dba 2025-10-10T01:23:15.1813981Z * [new tag] trunk/005c3d449e4c655d2eb0d76949a8cd41ce88f979 -> trunk/005c3d449e4c655d2eb0d76949a8cd41ce88f979 2025-10-10T01:23:15.1815576Z * [new tag] trunk/00f0365b959323bab89dc0a5bd5d40589e78edc8 -> trunk/00f0365b959323bab89dc0a5bd5d40589e78edc8 2025-10-10T01:23:15.1817186Z * [new tag] trunk/01f3a43462da594b65a6c9e8b46c132cd360cea9 -> trunk/01f3a43462da594b65a6c9e8b46c132cd360cea9 2025-10-10T01:23:15.1818580Z * [new tag] trunk/0319556a35b01e8857f7bf75df9df3287e1e853a -> trunk/0319556a35b01e8857f7bf75df9df3287e1e853a 2025-10-10T01:23:15.1819940Z * [new tag] trunk/054268c9ebb3291c6fd442e4a1f6602a8ea43ab6 -> trunk/054268c9ebb3291c6fd442e4a1f6602a8ea43ab6 2025-10-10T01:23:15.1821330Z * [new tag] trunk/06d86e58d0309aa2c217256f88d1990a22ec6e4f -> trunk/06d86e58d0309aa2c217256f88d1990a22ec6e4f 2025-10-10T01:23:15.1822807Z * [new tag] trunk/078d475d3bb104823e70ce975c2ee0d4d2fb0952 -> trunk/078d475d3bb104823e70ce975c2ee0d4d2fb0952 2025-10-10T01:23:15.1824205Z * [new tag] trunk/086dec3235d463e751c12ce9eeeb2dfcc873e206 -> trunk/086dec3235d463e751c12ce9eeeb2dfcc873e206 2025-10-10T01:23:15.1825756Z * [new tag] trunk/0a3e4e894cbc0cc93568c5d016f3ad72650cf641 -> trunk/0a3e4e894cbc0cc93568c5d016f3ad72650cf641 2025-10-10T01:23:15.1827170Z * [new tag] trunk/0b01ff4de02035eb21c1bc6bf4b1b627bc1cefaa -> trunk/0b01ff4de02035eb21c1bc6bf4b1b627bc1cefaa 2025-10-10T01:23:15.1828513Z * [new tag] trunk/0b15f7ae059cf4fa3909bbb009d83c0253a6385a -> trunk/0b15f7ae059cf4fa3909bbb009d83c0253a6385a 2025-10-10T01:23:15.1829914Z * [new tag] trunk/0b4f2b46d9e14c1858dd3d0ca9b62e349ae316cf -> trunk/0b4f2b46d9e14c1858dd3d0ca9b62e349ae316cf 2025-10-10T01:23:15.1831328Z * [new tag] trunk/0b85236477fe8a0e32510bcc973b2f34ef981df2 -> trunk/0b85236477fe8a0e32510bcc973b2f34ef981df2 2025-10-10T01:23:15.1832673Z * [new tag] trunk/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 -> trunk/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 2025-10-10T01:23:15.1834092Z * [new tag] trunk/0e5773b7fadef9e29b006af470b771fad55b5206 -> trunk/0e5773b7fadef9e29b006af470b771fad55b5206 2025-10-10T01:23:15.1835445Z * [new tag] trunk/0e9b3a772ab96e998ab85591d5b2a9c1d41bacb0 -> trunk/0e9b3a772ab96e998ab85591d5b2a9c1d41bacb0 2025-10-10T01:23:15.1836896Z * [new tag] trunk/0fbe3f19c7e88ee1720d2e1579e3fd2cafdaabf9 -> trunk/0fbe3f19c7e88ee1720d2e1579e3fd2cafdaabf9 2025-10-10T01:23:15.1838355Z * [new tag] trunk/0fd976b65c6daf3799a501d9202e4f50144446d1 -> trunk/0fd976b65c6daf3799a501d9202e4f50144446d1 2025-10-10T01:23:15.1839890Z * [new tag] trunk/1051c1de5c0c1d34bec94c4a3199ac7b23bb19e1 -> trunk/1051c1de5c0c1d34bec94c4a3199ac7b23bb19e1 2025-10-10T01:23:15.1841271Z * [new tag] trunk/115af42e9d57e89c26777be72822107cd7b39e07 -> trunk/115af42e9d57e89c26777be72822107cd7b39e07 2025-10-10T01:23:15.1842619Z * [new tag] trunk/11f5f656867089dac1fa1e64e34c9966578fbddd -> trunk/11f5f656867089dac1fa1e64e34c9966578fbddd 2025-10-10T01:23:15.1844071Z * [new tag] trunk/12d2ef557f6e127100267c31a31572d8ab5cc788 -> trunk/12d2ef557f6e127100267c31a31572d8ab5cc788 2025-10-10T01:23:15.1845348Z * [new tag] trunk/144378615a5a2b347e39c6376cba7d75f7a82926 -> trunk/144378615a5a2b347e39c6376cba7d75f7a82926 2025-10-10T01:23:15.1846734Z * [new tag] trunk/14791ea947349fb5fa7b7d6230cfd3924c36ba27 -> trunk/14791ea947349fb5fa7b7d6230cfd3924c36ba27 2025-10-10T01:23:15.1848169Z * [new tag] trunk/15800888b697bacd555399b3a0ca2e8d0827528e -> trunk/15800888b697bacd555399b3a0ca2e8d0827528e 2025-10-10T01:23:15.1849583Z * [new tag] trunk/15c8bdcc5e3a6dfd14e5c977438f772031e064ff -> trunk/15c8bdcc5e3a6dfd14e5c977438f772031e064ff 2025-10-10T01:23:15.1851066Z * [new tag] trunk/15d726005ddc5558c934c3edd5f815c2e504e501 -> trunk/15d726005ddc5558c934c3edd5f815c2e504e501 2025-10-10T01:23:15.1852419Z * [new tag] trunk/16f9bef642b07b3090a6e4a04517eff84d41a197 -> trunk/16f9bef642b07b3090a6e4a04517eff84d41a197 2025-10-10T01:23:15.1853916Z * [new tag] trunk/17c7170ca6e2efd5ead2b93bd12e226ff48f0669 -> trunk/17c7170ca6e2efd5ead2b93bd12e226ff48f0669 2025-10-10T01:23:15.1855405Z * [new tag] trunk/184817c7a81d5c01e107a84efeb269b063ddf5d6 -> trunk/184817c7a81d5c01e107a84efeb269b063ddf5d6 2025-10-10T01:23:15.1856881Z * [new tag] trunk/18940820006d2304460008575561e2e8e7fc59fc -> trunk/18940820006d2304460008575561e2e8e7fc59fc 2025-10-10T01:23:15.1858241Z * [new tag] trunk/18e18488e8c90e53cc113b1a5eddd9640ee80292 -> trunk/18e18488e8c90e53cc113b1a5eddd9640ee80292 2025-10-10T01:23:15.1859857Z * [new tag] trunk/1927783aa3ad676db6f4c34fc77ef3825a4e2ed5 -> trunk/1927783aa3ad676db6f4c34fc77ef3825a4e2ed5 2025-10-10T01:23:15.1861275Z * [new tag] trunk/19bf67be3286c0e2babe83af0d1593bae850362a -> trunk/19bf67be3286c0e2babe83af0d1593bae850362a 2025-10-10T01:23:15.1862678Z * [new tag] trunk/1bb68271b7ff1b582845384c6c7f7b1593ae1619 -> trunk/1bb68271b7ff1b582845384c6c7f7b1593ae1619 2025-10-10T01:23:15.1864035Z * [new tag] trunk/1d182dd81c3143697337e35d046fd02951dedb09 -> trunk/1d182dd81c3143697337e35d046fd02951dedb09 2025-10-10T01:23:15.1865381Z * [new tag] trunk/1e42fde45eff81845f269e8185f54a19f6d87c5b -> trunk/1e42fde45eff81845f269e8185f54a19f6d87c5b 2025-10-10T01:23:15.1866859Z * [new tag] trunk/1f73b96668bc6ae4c8e7ef5b630ff5f3c69ae005 -> trunk/1f73b96668bc6ae4c8e7ef5b630ff5f3c69ae005 2025-10-10T01:23:15.1868207Z * [new tag] trunk/1f8ee5da117952b03f0050a178d69f8e7189b0f8 -> trunk/1f8ee5da117952b03f0050a178d69f8e7189b0f8 2025-10-10T01:23:15.1869612Z * [new tag] trunk/1f9614cef8e0272c8e3bd99004d2978a6ecc5195 -> trunk/1f9614cef8e0272c8e3bd99004d2978a6ecc5195 2025-10-10T01:23:15.1870957Z * [new tag] trunk/1fb072ac2a33af93a77888dddbdd228b22a3f9c4 -> trunk/1fb072ac2a33af93a77888dddbdd228b22a3f9c4 2025-10-10T01:23:15.1872140Z * [new tag] trunk/1fc71d1b578badb1b3ba7cc2d5795f4f80463749 -> trunk/1fc71d1b578badb1b3ba7cc2d5795f4f80463749 2025-10-10T01:23:15.1873475Z * [new tag] trunk/20082d713666fa1eade588bebd523d86309bfa25 -> trunk/20082d713666fa1eade588bebd523d86309bfa25 2025-10-10T01:23:15.1874976Z * [new tag] trunk/2164b661219ab0a76aa018e955ba3d8e8f99c083 -> trunk/2164b661219ab0a76aa018e955ba3d8e8f99c083 2025-10-10T01:23:15.1876494Z * [new tag] trunk/228973df7f770505aafc6fc17b99f81ac58bdfe1 -> trunk/228973df7f770505aafc6fc17b99f81ac58bdfe1 2025-10-10T01:23:15.1877880Z * [new tag] trunk/22b1710252525d80d47ba95c762ccdbf577b2dc2 -> trunk/22b1710252525d80d47ba95c762ccdbf577b2dc2 2025-10-10T01:23:15.1879342Z * [new tag] trunk/22e219d9969ff3cee85bc5de32fa49d5a549a148 -> trunk/22e219d9969ff3cee85bc5de32fa49d5a549a148 2025-10-10T01:23:15.1880761Z * [new tag] trunk/235b995ce18de632ab816940319fcd66b46039b8 -> trunk/235b995ce18de632ab816940319fcd66b46039b8 2025-10-10T01:23:15.1881992Z * [new tag] trunk/23ab6a45e5c759fb4714905cb8c84ef74c70aa67 -> trunk/23ab6a45e5c759fb4714905cb8c84ef74c70aa67 2025-10-10T01:23:15.1884330Z * [new tag] trunk/24d69c57cbaa94cc828dbbdf83c889f5f244ae28 -> trunk/24d69c57cbaa94cc828dbbdf83c889f5f244ae28 2025-10-10T01:23:15.1885575Z * [new tag] trunk/263db92563f0ae71bf3e4fc265fbb48e79f9f23f -> trunk/263db92563f0ae71bf3e4fc265fbb48e79f9f23f 2025-10-10T01:23:15.1886475Z * [new tag] trunk/27234792add2ee9bedd84ca02dbf34f8f244bc5c -> trunk/27234792add2ee9bedd84ca02dbf34f8f244bc5c 2025-10-10T01:23:15.1887981Z * [new tag] trunk/27eb36debbe3fa2d43a2f893a5c46a6257a09460 -> trunk/27eb36debbe3fa2d43a2f893a5c46a6257a09460 2025-10-10T01:23:15.1889423Z * [new tag] trunk/2855a045b30dafad7a08d66e242be13770189c19 -> trunk/2855a045b30dafad7a08d66e242be13770189c19 2025-10-10T01:23:15.1890811Z * [new tag] trunk/2883b5ab773daf5861d43ff0b65be49a441ab3f9 -> trunk/2883b5ab773daf5861d43ff0b65be49a441ab3f9 2025-10-10T01:23:15.1892147Z * [new tag] trunk/29824067215f3ed9e4044ca0f31a71e9d95f237d -> trunk/29824067215f3ed9e4044ca0f31a71e9d95f237d 2025-10-10T01:23:15.1893781Z * [new tag] trunk/2a11ce2c787b2339ffb8941b849dd487d25b4121 -> trunk/2a11ce2c787b2339ffb8941b849dd487d25b4121 2025-10-10T01:23:15.1895008Z * [new tag] trunk/2a6cdba6e5f74c2294fecc2d1344537522efbaab -> trunk/2a6cdba6e5f74c2294fecc2d1344537522efbaab 2025-10-10T01:23:15.1896626Z * [new tag] trunk/2a760dc51e04d65845440cc09e7016cfc74f9132 -> trunk/2a760dc51e04d65845440cc09e7016cfc74f9132 2025-10-10T01:23:15.1898171Z * [new tag] trunk/2a7c48675010056f23d62b5c6ecb318782801723 -> trunk/2a7c48675010056f23d62b5c6ecb318782801723 2025-10-10T01:23:15.1899608Z * [new tag] trunk/2b58adc3bdcf9476e1cef49ad965b7d3c7b9ac24 -> trunk/2b58adc3bdcf9476e1cef49ad965b7d3c7b9ac24 2025-10-10T01:23:15.1900954Z * [new tag] trunk/2b9ff9953523a2e916234c9197d946f4cff976c7 -> trunk/2b9ff9953523a2e916234c9197d946f4cff976c7 2025-10-10T01:23:15.1902429Z * [new tag] trunk/2c2e1268b7aae8ed610d12f2d38d39f8d93888a3 -> trunk/2c2e1268b7aae8ed610d12f2d38d39f8d93888a3 2025-10-10T01:23:15.1903851Z * [new tag] trunk/2c5ed6e7c067573b093725cd15d13812d9647562 -> trunk/2c5ed6e7c067573b093725cd15d13812d9647562 2025-10-10T01:23:15.1905307Z * [new tag] trunk/2d50678dcc7ab2da13a9bca6af8f2333e8970344 -> trunk/2d50678dcc7ab2da13a9bca6af8f2333e8970344 2025-10-10T01:23:15.1906700Z * [new tag] trunk/2e027e874232fefe7b1c56ce8aeb26c0e6b97f15 -> trunk/2e027e874232fefe7b1c56ce8aeb26c0e6b97f15 2025-10-10T01:23:15.1908079Z * [new tag] trunk/2e1742dd63c2168fd9649dbba96a95abf1f57cae -> trunk/2e1742dd63c2168fd9649dbba96a95abf1f57cae 2025-10-10T01:23:15.1909507Z * [new tag] trunk/2fe37b5fde392535a3238f975c93dd202cd3e24b -> trunk/2fe37b5fde392535a3238f975c93dd202cd3e24b 2025-10-10T01:23:15.1910920Z * [new tag] trunk/3040a5d294bd30d3938d0043a5d93d6c23264827 -> trunk/3040a5d294bd30d3938d0043a5d93d6c23264827 2025-10-10T01:23:15.1912363Z * [new tag] trunk/321e6026925f6b6e8a36e3a8b7c0295cd7541911 -> trunk/321e6026925f6b6e8a36e3a8b7c0295cd7541911 2025-10-10T01:23:15.1913633Z * [new tag] trunk/322091d8d8542a0cbff524306029bef4d7338747 -> trunk/322091d8d8542a0cbff524306029bef4d7338747 2025-10-10T01:23:15.1915052Z * [new tag] trunk/3288fbf374128610928e27d03615ac0d46a6ce14 -> trunk/3288fbf374128610928e27d03615ac0d46a6ce14 2025-10-10T01:23:15.1916447Z * [new tag] trunk/331191ce4b29b5d7d3bb7f0e7454ca70c06fbd26 -> trunk/331191ce4b29b5d7d3bb7f0e7454ca70c06fbd26 2025-10-10T01:23:15.1917786Z * [new tag] trunk/33b17bc619b044a0050797987efb8890d43319df -> trunk/33b17bc619b044a0050797987efb8890d43319df 2025-10-10T01:23:15.1919382Z * [new tag] trunk/34042a9145fe28033e7edb08f1fcf90ed197f4ac -> trunk/34042a9145fe28033e7edb08f1fcf90ed197f4ac 2025-10-10T01:23:15.1920798Z * [new tag] trunk/344e6365a0068c2d2847fcec0c55dd53291d475e -> trunk/344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:23:15.1922292Z * [new tag] trunk/34ac9b61cbfcf17328ccb8b729509829447fdddd -> trunk/34ac9b61cbfcf17328ccb8b729509829447fdddd 2025-10-10T01:23:15.1923777Z * [new tag] trunk/35c4130fd1358c98e12301ffa0f1b2294e0c795f -> trunk/35c4130fd1358c98e12301ffa0f1b2294e0c795f 2025-10-10T01:23:15.1925007Z * [new tag] trunk/35f66b83f89a571d0c0abe16c66a23120b92bdaf -> trunk/35f66b83f89a571d0c0abe16c66a23120b92bdaf 2025-10-10T01:23:15.1926294Z * [new tag] trunk/361c5d362c4ea1950e05116899cfcf753c345ebd -> trunk/361c5d362c4ea1950e05116899cfcf753c345ebd 2025-10-10T01:23:15.1927713Z * [new tag] trunk/37c6087334cce3ad4bc9838ea2ef63aba89f2253 -> trunk/37c6087334cce3ad4bc9838ea2ef63aba89f2253 2025-10-10T01:23:15.1929109Z * [new tag] trunk/3912ba3e940b9354622fa09b2ada677cd10723d8 -> trunk/3912ba3e940b9354622fa09b2ada677cd10723d8 2025-10-10T01:23:15.1930537Z * [new tag] trunk/39189592fd688979e56063430ed5a038d999908f -> trunk/39189592fd688979e56063430ed5a038d999908f 2025-10-10T01:23:15.1932057Z * [new tag] trunk/3924f784ba81f87fe09988d6fc9620b57e4d9f72 -> trunk/3924f784ba81f87fe09988d6fc9620b57e4d9f72 2025-10-10T01:23:15.1933394Z * [new tag] trunk/39b31a6bfde6e046383ae2b06fe0b68df5cdbdd2 -> trunk/39b31a6bfde6e046383ae2b06fe0b68df5cdbdd2 2025-10-10T01:23:15.1934754Z * [new tag] trunk/39c340ec9e2ee3011f1d260f581b5a95f3c99039 -> trunk/39c340ec9e2ee3011f1d260f581b5a95f3c99039 2025-10-10T01:23:15.1936144Z * [new tag] trunk/39d0c06ed0d7bc634d7f1a4e84b69f66d1ea0798 -> trunk/39d0c06ed0d7bc634d7f1a4e84b69f66d1ea0798 2025-10-10T01:23:15.1937523Z * [new tag] trunk/3c0577bd15778c96cecf0e7a5e5958d7fcab64f0 -> trunk/3c0577bd15778c96cecf0e7a5e5958d7fcab64f0 2025-10-10T01:23:15.1938931Z * [new tag] trunk/3c59351c6ea2fc29d346903e28e95c5f4d0ccdbb -> trunk/3c59351c6ea2fc29d346903e28e95c5f4d0ccdbb 2025-10-10T01:23:15.1940349Z * [new tag] trunk/3c5ca685d6f5b6f3971c0cd20a054aa355610419 -> trunk/3c5ca685d6f5b6f3971c0cd20a054aa355610419 2025-10-10T01:23:15.1941849Z * [new tag] trunk/3ca09d65f1bdf83142dc9fe47976227ae4a88e7b -> trunk/3ca09d65f1bdf83142dc9fe47976227ae4a88e7b 2025-10-10T01:23:15.1943445Z * [new tag] trunk/3cc8af2d67f42bf2a933796290446c5ab8978aac -> trunk/3cc8af2d67f42bf2a933796290446c5ab8978aac 2025-10-10T01:23:15.1944832Z * [new tag] trunk/3d1fa40ae1fee18ddf3dca89229e3ae828589e0c -> trunk/3d1fa40ae1fee18ddf3dca89229e3ae828589e0c 2025-10-10T01:23:15.1946224Z * [new tag] trunk/3d9d41c80168bcd3c569345a96682c42a5eba36a -> trunk/3d9d41c80168bcd3c569345a96682c42a5eba36a 2025-10-10T01:23:15.1947588Z * [new tag] trunk/3db21643417a04f6f2707a783ac32a538a98d53d -> trunk/3db21643417a04f6f2707a783ac32a538a98d53d 2025-10-10T01:23:15.1948973Z * [new tag] trunk/3ddf2018d0b7b4def0553dc092d928ef831a19c3 -> trunk/3ddf2018d0b7b4def0553dc092d928ef831a19c3 2025-10-10T01:23:15.1950359Z * [new tag] trunk/3e03deab6f3c268c85c8efd9546e28cdda0fa4cc -> trunk/3e03deab6f3c268c85c8efd9546e28cdda0fa4cc 2025-10-10T01:23:15.1951832Z * [new tag] trunk/3e0826c9d792ae87373dc0ff5d46c260020de29f -> trunk/3e0826c9d792ae87373dc0ff5d46c260020de29f 2025-10-10T01:23:15.1953212Z * [new tag] trunk/409aece3f9436a2f740f1b97f1243f738f6bbbf6 -> trunk/409aece3f9436a2f740f1b97f1243f738f6bbbf6 2025-10-10T01:23:15.1954580Z * [new tag] trunk/40b25578e4ecb7ef1c38201b3ce0014eb57c53eb -> trunk/40b25578e4ecb7ef1c38201b3ce0014eb57c53eb 2025-10-10T01:23:15.1955856Z * [new tag] trunk/412c6d28ec3869ef8ba962b290d755251e7cc3c1 -> trunk/412c6d28ec3869ef8ba962b290d755251e7cc3c1 2025-10-10T01:23:15.1957122Z * [new tag] trunk/415e641572473479fc9d9eaea12762e1a223a9e0 -> trunk/415e641572473479fc9d9eaea12762e1a223a9e0 2025-10-10T01:23:15.1958543Z * [new tag] trunk/41808b2ba9a61ab2f4c7af394c1668d09a4a0331 -> trunk/41808b2ba9a61ab2f4c7af394c1668d09a4a0331 2025-10-10T01:23:15.1960176Z * [new tag] trunk/4308b8a28fa332d23ad6d25a472559b354619131 -> trunk/4308b8a28fa332d23ad6d25a472559b354619131 2025-10-10T01:23:15.1961385Z * [new tag] trunk/43848b71d9af0223eafdd1755bf7444aafe9e993 -> trunk/43848b71d9af0223eafdd1755bf7444aafe9e993 2025-10-10T01:23:15.1962820Z * [new tag] trunk/43fc859625f9c0a794307b3ef30c26ab3fc2bfec -> trunk/43fc859625f9c0a794307b3ef30c26ab3fc2bfec 2025-10-10T01:23:15.1964286Z * [new tag] trunk/4412026949b562f940d4c24162de19d299725b62 -> trunk/4412026949b562f940d4c24162de19d299725b62 2025-10-10T01:23:15.1965676Z * [new tag] trunk/44a5d419935a77b3308f247279a457e6d0b9a292 -> trunk/44a5d419935a77b3308f247279a457e6d0b9a292 2025-10-10T01:23:15.1967125Z * [new tag] trunk/4661200125ba9c87aa7d54a55e585403b5ce5040 -> trunk/4661200125ba9c87aa7d54a55e585403b5ce5040 2025-10-10T01:23:15.1968564Z * [new tag] trunk/4691fe60700ac51a878775fd23a8f7c4548c6757 -> trunk/4691fe60700ac51a878775fd23a8f7c4548c6757 2025-10-10T01:23:15.1970020Z * [new tag] trunk/4725871a815fb880e89135a493c8c94ab9bbfece -> trunk/4725871a815fb880e89135a493c8c94ab9bbfece 2025-10-10T01:23:15.1971396Z * [new tag] trunk/47956196d99166fe9083beb2a52fd2e6c90b2011 -> trunk/47956196d99166fe9083beb2a52fd2e6c90b2011 2025-10-10T01:23:15.1972884Z * [new tag] trunk/483f4e0db91166128ad8922d86dc7222338d4ecc -> trunk/483f4e0db91166128ad8922d86dc7222338d4ecc 2025-10-10T01:23:15.1974320Z * [new tag] trunk/48b54b45d62af7ecafccc5afede04474cb236f1a -> trunk/48b54b45d62af7ecafccc5afede04474cb236f1a 2025-10-10T01:23:15.1976048Z * [new tag] trunk/49f7d8d19d24f616b11ef050535a211245aed649 -> trunk/49f7d8d19d24f616b11ef050535a211245aed649 2025-10-10T01:23:15.1977461Z * [new tag] trunk/4a0df39f814afad087e8b29dd2914a8b54567694 -> trunk/4a0df39f814afad087e8b29dd2914a8b54567694 2025-10-10T01:23:15.1978716Z * [new tag] trunk/4a6abba0d9fb3dc0f29b5efe527e26b2962caec1 -> trunk/4a6abba0d9fb3dc0f29b5efe527e26b2962caec1 2025-10-10T01:23:15.1980207Z * [new tag] trunk/4ab847bbc7ba09f29a4e81494e8a752dcb411117 -> trunk/4ab847bbc7ba09f29a4e81494e8a752dcb411117 2025-10-10T01:23:15.1981562Z * [new tag] trunk/4bcc05777e780e834d44a2d06dd5321daec316f0 -> trunk/4bcc05777e780e834d44a2d06dd5321daec316f0 2025-10-10T01:23:15.1983214Z * [new tag] trunk/4bd1505f849e701a8e54f9d185c23f13e7324498 -> trunk/4bd1505f849e701a8e54f9d185c23f13e7324498 2025-10-10T01:23:15.1984514Z * [new tag] trunk/4c0fec3e4dac35b9e9dec2beacfb5967906a4701 -> trunk/4c0fec3e4dac35b9e9dec2beacfb5967906a4701 2025-10-10T01:23:15.1985957Z * [new tag] trunk/4c3c0ef2f10415f5d5b13f1c842f91bb90ee91d3 -> trunk/4c3c0ef2f10415f5d5b13f1c842f91bb90ee91d3 2025-10-10T01:23:15.1987347Z * [new tag] trunk/4d7f9f3aed68729380730ed46e29ff2052f05b73 -> trunk/4d7f9f3aed68729380730ed46e29ff2052f05b73 2025-10-10T01:23:15.1988767Z * [new tag] trunk/50e077beaaf71798f870552f3849e4a52c784df5 -> trunk/50e077beaaf71798f870552f3849e4a52c784df5 2025-10-10T01:23:15.1990198Z * [new tag] trunk/5103ecc5d8f0cc90e686763652e2d84c22d83ca9 -> trunk/5103ecc5d8f0cc90e686763652e2d84c22d83ca9 2025-10-10T01:23:15.1991803Z * [new tag] trunk/5178d0a480f8f4e21da3757de455c8215b249ec5 -> trunk/5178d0a480f8f4e21da3757de455c8215b249ec5 2025-10-10T01:23:15.1993422Z * [new tag] trunk/5209c8ce0704f34ba4bd2a58c19877fbf6cf0392 -> trunk/5209c8ce0704f34ba4bd2a58c19877fbf6cf0392 2025-10-10T01:23:15.1994912Z * [new tag] trunk/5390324984c43f1214b8abf731ad495ba2df5341 -> trunk/5390324984c43f1214b8abf731ad495ba2df5341 2025-10-10T01:23:15.1996574Z * [new tag] trunk/53f5af8c924aba3c0fab1fabc6baf7d6affcb8a1 -> trunk/53f5af8c924aba3c0fab1fabc6baf7d6affcb8a1 2025-10-10T01:23:15.1998003Z * [new tag] trunk/54ae61c573e91fa2a2c6430435059e2d94ecba2e -> trunk/54ae61c573e91fa2a2c6430435059e2d94ecba2e 2025-10-10T01:23:15.1999644Z * [new tag] trunk/5656d45c8ff03cf20fd7d5098247c2250395af8a -> trunk/5656d45c8ff03cf20fd7d5098247c2250395af8a 2025-10-10T01:23:15.2000962Z * [new tag] trunk/56d66ac0d74f44d7b656757795142b5b9a1802a1 -> trunk/56d66ac0d74f44d7b656757795142b5b9a1802a1 2025-10-10T01:23:15.2002301Z * [new tag] trunk/5743d731c1de495ecf3bb03682a2dcbe207ca895 -> trunk/5743d731c1de495ecf3bb03682a2dcbe207ca895 2025-10-10T01:23:15.2003786Z * [new tag] trunk/5a1fbf45ad727353e367740ecd8825ca7ee857e9 -> trunk/5a1fbf45ad727353e367740ecd8825ca7ee857e9 2025-10-10T01:23:15.2005180Z * [new tag] trunk/5a66ff4915ecfd86f1a68e7862e5a2ad473e5a79 -> trunk/5a66ff4915ecfd86f1a68e7862e5a2ad473e5a79 2025-10-10T01:23:15.2006599Z * [new tag] trunk/5b0b4cda4aa03bee16ee67d9d36012a539df3c50 -> trunk/5b0b4cda4aa03bee16ee67d9d36012a539df3c50 2025-10-10T01:23:15.2008083Z * [new tag] trunk/5b8174bc286725f9326fba6dc0ef17c316486bbd -> trunk/5b8174bc286725f9326fba6dc0ef17c316486bbd 2025-10-10T01:23:15.2009469Z * [new tag] trunk/5ba11df4f871717818b88c4eab514d31286601d1 -> trunk/5ba11df4f871717818b88c4eab514d31286601d1 2025-10-10T01:23:15.2010640Z * [new tag] trunk/5c827a4133da69108338d0363bb7ad7f62803c40 -> trunk/5c827a4133da69108338d0363bb7ad7f62803c40 2025-10-10T01:23:15.2012132Z * [new tag] trunk/5d459dd6099ef94d33db9a6d36bcce9f742f1da1 -> trunk/5d459dd6099ef94d33db9a6d36bcce9f742f1da1 2025-10-10T01:23:15.2013402Z * [new tag] trunk/5d7360bb03355c89c0b956df0ab428f5a7b5c9f8 -> trunk/5d7360bb03355c89c0b956df0ab428f5a7b5c9f8 2025-10-10T01:23:15.2015186Z * [new tag] trunk/5dbae1eae26159058f6199fc68fe73fc0e5bef5f -> trunk/5dbae1eae26159058f6199fc68fe73fc0e5bef5f 2025-10-10T01:23:15.2016508Z * [new tag] trunk/5e47b4dd60ff9efb253286af5a2479d9d800ce6a -> trunk/5e47b4dd60ff9efb253286af5a2479d9d800ce6a 2025-10-10T01:23:15.2017999Z * [new tag] trunk/5ed4270440fd0b62d3aa14692f9e377a0061061e -> trunk/5ed4270440fd0b62d3aa14692f9e377a0061061e 2025-10-10T01:23:15.2019499Z * [new tag] trunk/5f18f240de43fc24481ead4d740dda64f174fa86 -> trunk/5f18f240de43fc24481ead4d740dda64f174fa86 2025-10-10T01:23:15.2020938Z * [new tag] trunk/5f775bdfb766d9a2717ffbb64f2a51e53cddc778 -> trunk/5f775bdfb766d9a2717ffbb64f2a51e53cddc778 2025-10-10T01:23:15.2022206Z * [new tag] trunk/600267ea56cafcf8f9a1150a4379184960a757b2 -> trunk/600267ea56cafcf8f9a1150a4379184960a757b2 2025-10-10T01:23:15.2023684Z * [new tag] trunk/600db525bdb5e76c12f30f271d969d43a7f8efef -> trunk/600db525bdb5e76c12f30f271d969d43a7f8efef 2025-10-10T01:23:15.2025250Z * [new tag] trunk/608792153f42254d2d2b5a87d524807a0c2724f1 -> trunk/608792153f42254d2d2b5a87d524807a0c2724f1 2025-10-10T01:23:15.2026766Z * [new tag] trunk/6389658ec6b1ea58cb1de032266d865eeb8d48e9 -> trunk/6389658ec6b1ea58cb1de032266d865eeb8d48e9 2025-10-10T01:23:15.2028234Z * [new tag] trunk/64108bdbed2f099d527060b4c9fdd5a11cad2afc -> trunk/64108bdbed2f099d527060b4c9fdd5a11cad2afc 2025-10-10T01:23:15.2029657Z * [new tag] trunk/65aa62d50d1c83aa1b46ed4d584f12f509bab1c4 -> trunk/65aa62d50d1c83aa1b46ed4d584f12f509bab1c4 2025-10-10T01:23:15.2031120Z * [new tag] trunk/65f10becdf21f3a0947a735904fcce876ce3c4b0 -> trunk/65f10becdf21f3a0947a735904fcce876ce3c4b0 2025-10-10T01:23:15.2032524Z * [new tag] trunk/660e369a68dd8be60ce4eb67c25191ea66efc303 -> trunk/660e369a68dd8be60ce4eb67c25191ea66efc303 2025-10-10T01:23:15.2033973Z * [new tag] trunk/68350660ee2db8c21c84527929b92de9f0bcc3e2 -> trunk/68350660ee2db8c21c84527929b92de9f0bcc3e2 2025-10-10T01:23:15.2035391Z * [new tag] trunk/6861a270624b44954826688f8dad668eb0154452 -> trunk/6861a270624b44954826688f8dad668eb0154452 2025-10-10T01:23:15.2037021Z * [new tag] trunk/6861fa43e5fee7fedc0213e352fa983edea8aa78 -> trunk/6861fa43e5fee7fedc0213e352fa983edea8aa78 2025-10-10T01:23:15.2038364Z * [new tag] trunk/688efd9741dbd18c176729aec3df7a73825f8463 -> trunk/688efd9741dbd18c176729aec3df7a73825f8463 2025-10-10T01:23:15.2039852Z * [new tag] trunk/6a09f9306cadd003b2e6abc3f6422a2d8607779b -> trunk/6a09f9306cadd003b2e6abc3f6422a2d8607779b 2025-10-10T01:23:15.2041613Z * [new tag] trunk/6a31f42da45c0f1cbdb021b3695f0e6388b8b532 -> trunk/6a31f42da45c0f1cbdb021b3695f0e6388b8b532 2025-10-10T01:23:15.2043110Z * [new tag] trunk/6a7f5c0d21a22959d014c8b06f3efe3408336aaf -> trunk/6a7f5c0d21a22959d014c8b06f3efe3408336aaf 2025-10-10T01:23:15.2044645Z * [new tag] trunk/6b768e1890a179122e91395c5532a382d69b96a0 -> trunk/6b768e1890a179122e91395c5532a382d69b96a0 2025-10-10T01:23:15.2046195Z * [new tag] trunk/6b7970192f5de47d29a4fe085f509389ac0bea7d -> trunk/6b7970192f5de47d29a4fe085f509389ac0bea7d 2025-10-10T01:23:15.2047704Z * [new tag] trunk/6bb021c12553755a4f64df0b60dc34b1efdb992b -> trunk/6bb021c12553755a4f64df0b60dc34b1efdb992b 2025-10-10T01:23:15.2049101Z * [new tag] trunk/6bb586eafd723d4972c729f37c14f27c88168adc -> trunk/6bb586eafd723d4972c729f37c14f27c88168adc 2025-10-10T01:23:15.2050601Z * [new tag] trunk/6c0125dbc0241aef962528651df4f67204a8b526 -> trunk/6c0125dbc0241aef962528651df4f67204a8b526 2025-10-10T01:23:15.2052015Z * [new tag] trunk/6c209bfc5c1e1e59e6a62f94151398d66164bb93 -> trunk/6c209bfc5c1e1e59e6a62f94151398d66164bb93 2025-10-10T01:23:15.2053512Z * [new tag] trunk/6c3c9414eb571b34ff0d932978e4733dbb08dc1d -> trunk/6c3c9414eb571b34ff0d932978e4733dbb08dc1d 2025-10-10T01:23:15.2054951Z * [new tag] trunk/6d27a8e5093ee2a21d44dceeeffcb272e6e0f655 -> trunk/6d27a8e5093ee2a21d44dceeeffcb272e6e0f655 2025-10-10T01:23:15.2056469Z * [new tag] trunk/702f6e703b1d3a942346848b65a9f2a37d12ae18 -> trunk/702f6e703b1d3a942346848b65a9f2a37d12ae18 2025-10-10T01:23:15.2057937Z * [new tag] trunk/7158aa22e8dc97fdc2657cf0d4cde34b277e7d9e -> trunk/7158aa22e8dc97fdc2657cf0d4cde34b277e7d9e 2025-10-10T01:23:15.2059408Z * [new tag] trunk/71aefd5595834dd97f38aa978ee32abbd13ac3d6 -> trunk/71aefd5595834dd97f38aa978ee32abbd13ac3d6 2025-10-10T01:23:15.2060840Z * [new tag] trunk/724463d5a2fba369cd14e89215b84d1b01435df7 -> trunk/724463d5a2fba369cd14e89215b84d1b01435df7 2025-10-10T01:23:15.2062112Z * [new tag] trunk/73adac05d13babb75410c3e033fdce57aa16881a -> trunk/73adac05d13babb75410c3e033fdce57aa16881a 2025-10-10T01:23:15.2063563Z * [new tag] trunk/7457d139c51124e5a31a6173f99f81f0deb52178 -> trunk/7457d139c51124e5a31a6173f99f81f0deb52178 2025-10-10T01:23:15.2065000Z * [new tag] trunk/746fe78ecd52f3e9cfddda41f0ac82dada7bdd0b -> trunk/746fe78ecd52f3e9cfddda41f0ac82dada7bdd0b 2025-10-10T01:23:15.2066434Z * [new tag] trunk/7614338b69481d702c9f084ac15d9109c7cd3ef0 -> trunk/7614338b69481d702c9f084ac15d9109c7cd3ef0 2025-10-10T01:23:15.2067751Z * [new tag] trunk/7617b113ad0045cdfe5cf1feb8efb634a41c6ce2 -> trunk/7617b113ad0045cdfe5cf1feb8efb634a41c6ce2 2025-10-10T01:23:15.2069657Z * [new tag] trunk/7a1ead755f2e2abe8be49a7a0fb88b6b13973147 -> trunk/7a1ead755f2e2abe8be49a7a0fb88b6b13973147 2025-10-10T01:23:15.2071075Z * [new tag] trunk/7b15534434aeaf59a4c9189f52b4ebd4a5d58803 -> trunk/7b15534434aeaf59a4c9189f52b4ebd4a5d58803 2025-10-10T01:23:15.2072522Z * [new tag] trunk/7b691546d2949790ffc8f6bd3c674faa6a46ff7c -> trunk/7b691546d2949790ffc8f6bd3c674faa6a46ff7c 2025-10-10T01:23:15.2074082Z * [new tag] trunk/7cfecd76b2141d81c90d722dc5e3262bdf7ea900 -> trunk/7cfecd76b2141d81c90d722dc5e3262bdf7ea900 2025-10-10T01:23:15.2075603Z * [new tag] trunk/7d570129e0cea8dd3de0175baff96723656ab8ab -> trunk/7d570129e0cea8dd3de0175baff96723656ab8ab 2025-10-10T01:23:15.2076973Z * [new tag] trunk/7e7ac2039d5d5f35373c4de6cdf0ccdee3734c7a -> trunk/7e7ac2039d5d5f35373c4de6cdf0ccdee3734c7a 2025-10-10T01:23:15.2078311Z * [new tag] trunk/7eb1eb4313cfa3db1beadc6d9d04ea6b76acc39c -> trunk/7eb1eb4313cfa3db1beadc6d9d04ea6b76acc39c 2025-10-10T01:23:15.2079669Z * [new tag] trunk/801e282f39e9ef4424dfd3ecfd2b550a44595229 -> trunk/801e282f39e9ef4424dfd3ecfd2b550a44595229 2025-10-10T01:23:15.2081166Z * [new tag] trunk/81994b08a078b30e076d408713f78c9bf4e329e7 -> trunk/81994b08a078b30e076d408713f78c9bf4e329e7 2025-10-10T01:23:15.2082447Z * [new tag] trunk/81dbeb06f4b3eb6c56625ec25d377eb7c7c6c573 -> trunk/81dbeb06f4b3eb6c56625ec25d377eb7c7c6c573 2025-10-10T01:23:15.2083892Z * [new tag] trunk/83458197d14921f797565135f0f45031c362338d -> trunk/83458197d14921f797565135f0f45031c362338d 2025-10-10T01:23:15.2085341Z * [new tag] trunk/83d71dfb2fd993a6242372b8123549acaa85ffdb -> trunk/83d71dfb2fd993a6242372b8123549acaa85ffdb 2025-10-10T01:23:15.2086767Z * [new tag] trunk/86474ce996d168b404592cbbdfcc30d6607c8bd4 -> trunk/86474ce996d168b404592cbbdfcc30d6607c8bd4 2025-10-10T01:23:15.2088215Z * [new tag] trunk/86c789849eac1f96d03cf273e7995dbc7d319c26 -> trunk/86c789849eac1f96d03cf273e7995dbc7d319c26 2025-10-10T01:23:15.2089914Z * [new tag] trunk/874efa2d72d83b00894097130f18062ce331a265 -> trunk/874efa2d72d83b00894097130f18062ce331a265 2025-10-10T01:23:15.2091259Z * [new tag] trunk/87c9fbda22c229d4e5512011e050efd6ffea1241 -> trunk/87c9fbda22c229d4e5512011e050efd6ffea1241 2025-10-10T01:23:15.2092581Z * [new tag] trunk/87eccf10e8484c9e59ef81ae7bdee68d3db4f605 -> trunk/87eccf10e8484c9e59ef81ae7bdee68d3db4f605 2025-10-10T01:23:15.2094019Z * [new tag] trunk/8c0bc879b97bc580aaa0777b2d266bdd068cb528 -> trunk/8c0bc879b97bc580aaa0777b2d266bdd068cb528 2025-10-10T01:23:15.2095524Z * [new tag] trunk/8c54101933bb7c6ed3f9c1a65629b7f30376f7e2 -> trunk/8c54101933bb7c6ed3f9c1a65629b7f30376f7e2 2025-10-10T01:23:15.2097198Z * [new tag] trunk/8ca986ee60febce075f9e3ff83726048cebbbf68 -> trunk/8ca986ee60febce075f9e3ff83726048cebbbf68 2025-10-10T01:23:15.2098557Z * [new tag] trunk/8d53d788fefc0370931063d91f0c342556c3cf4c -> trunk/8d53d788fefc0370931063d91f0c342556c3cf4c 2025-10-10T01:23:15.2100672Z * [new tag] trunk/8e1f409b8ccf64b2cf3933ece13587ad57e9d8a9 -> trunk/8e1f409b8ccf64b2cf3933ece13587ad57e9d8a9 2025-10-10T01:23:15.2102307Z * [new tag] trunk/8ec8c14aced9f3e7ff4ab663822bed792d6c34f4 -> trunk/8ec8c14aced9f3e7ff4ab663822bed792d6c34f4 2025-10-10T01:23:15.2103751Z * [new tag] trunk/8f54e27e5decf41222f5d744069eb6572dbf275f -> trunk/8f54e27e5decf41222f5d744069eb6572dbf275f 2025-10-10T01:23:15.2105230Z * [new tag] trunk/8f705d019a64b1ca882e043b3eb98559273a9e59 -> trunk/8f705d019a64b1ca882e043b3eb98559273a9e59 2025-10-10T01:23:15.2106705Z * [new tag] trunk/8f83b3e71cb2af6244971af59bfbb6e2abb55f24 -> trunk/8f83b3e71cb2af6244971af59bfbb6e2abb55f24 2025-10-10T01:23:15.2108158Z * [new tag] trunk/90b4e130d6871bee4e1f15bb8294c1bbbf8f4ba5 -> trunk/90b4e130d6871bee4e1f15bb8294c1bbbf8f4ba5 2025-10-10T01:23:15.2109456Z * [new tag] trunk/90c0825e2deb0a46faf5cc2deb7184f6f8ea7a6d -> trunk/90c0825e2deb0a46faf5cc2deb7184f6f8ea7a6d 2025-10-10T01:23:15.2110920Z * [new tag] trunk/91040f49348646d79c6cd3434c34860d25c2e47a -> trunk/91040f49348646d79c6cd3434c34860d25c2e47a 2025-10-10T01:23:15.2112178Z * [new tag] trunk/91b94842645c1a781ab169b0df718545901ebb01 -> trunk/91b94842645c1a781ab169b0df718545901ebb01 2025-10-10T01:23:15.2113632Z * [new tag] trunk/91c211fb8c8ec3065be2a18dfc399ce849ea83bf -> trunk/91c211fb8c8ec3065be2a18dfc399ce849ea83bf 2025-10-10T01:23:15.2115296Z * [new tag] trunk/91c4db76cbb82dfa46d937b8dce4c942eaf5e226 -> trunk/91c4db76cbb82dfa46d937b8dce4c942eaf5e226 2025-10-10T01:23:15.2116654Z * [new tag] trunk/93e833de0f987f66d8c93b76ffe6aad35b714231 -> trunk/93e833de0f987f66d8c93b76ffe6aad35b714231 2025-10-10T01:23:15.2118044Z * [new tag] trunk/94b1ec8c7c5cc63541325abc923973f2fc2ad094 -> trunk/94b1ec8c7c5cc63541325abc923973f2fc2ad094 2025-10-10T01:23:15.2119623Z * [new tag] trunk/955f21dc2c628e09e0d112b3db1ee928cd1da344 -> trunk/955f21dc2c628e09e0d112b3db1ee928cd1da344 2025-10-10T01:23:15.2121242Z * [new tag] trunk/9580539e2f73d68e89544c713ff460bea3038701 -> trunk/9580539e2f73d68e89544c713ff460bea3038701 2025-10-10T01:23:15.2122686Z * [new tag] trunk/95a053284cd28e8d52bd55049bd45aea47adba0c -> trunk/95a053284cd28e8d52bd55049bd45aea47adba0c 2025-10-10T01:23:15.2124241Z * [new tag] trunk/960c4b9937251da01ea588efff0fc06a34eac35b -> trunk/960c4b9937251da01ea588efff0fc06a34eac35b 2025-10-10T01:23:15.2125497Z * [new tag] trunk/96181d6f7619acf938dc743123326c6b5dd25284 -> trunk/96181d6f7619acf938dc743123326c6b5dd25284 2025-10-10T01:23:15.2127014Z * [new tag] trunk/9697a7ce9ea095e933658cfee13f9bbef272551a -> trunk/9697a7ce9ea095e933658cfee13f9bbef272551a 2025-10-10T01:23:15.2128514Z * [new tag] trunk/96d91da792d4b50930318ecdfb8b5b8190c467cd -> trunk/96d91da792d4b50930318ecdfb8b5b8190c467cd 2025-10-10T01:23:15.2129972Z * [new tag] trunk/97463d4cf3c125557ef23502772b12a67dac4dc7 -> trunk/97463d4cf3c125557ef23502772b12a67dac4dc7 2025-10-10T01:23:15.2131416Z * [new tag] trunk/97ca21106d0179f425fc752ec867fe11669c2834 -> trunk/97ca21106d0179f425fc752ec867fe11669c2834 2025-10-10T01:23:15.2132909Z * [new tag] trunk/98a081a24c22072362dc536afd39a469e28939d4 -> trunk/98a081a24c22072362dc536afd39a469e28939d4 2025-10-10T01:23:15.2134372Z * [new tag] trunk/9944cac6e6a95159744a775a8bef40d89eef0f03 -> trunk/9944cac6e6a95159744a775a8bef40d89eef0f03 2025-10-10T01:23:15.2135813Z * [new tag] trunk/9aa92f246fa5fe5cfda17970d41d167b19a0612a -> trunk/9aa92f246fa5fe5cfda17970d41d167b19a0612a 2025-10-10T01:23:15.2137307Z * [new tag] trunk/9d1ab4f4bb508a72c7f549f0b5219c4601944ba1 -> trunk/9d1ab4f4bb508a72c7f549f0b5219c4601944ba1 2025-10-10T01:23:15.2138770Z * [new tag] trunk/9eb89a4ad5965b97c54e498d71fc765c0059acef -> trunk/9eb89a4ad5965b97c54e498d71fc765c0059acef 2025-10-10T01:23:15.2140133Z * [new tag] trunk/9ec10dc26a81dc618ff435edd4ca4819245ecb0f -> trunk/9ec10dc26a81dc618ff435edd4ca4819245ecb0f 2025-10-10T01:23:15.2141608Z * [new tag] trunk/9ecd092bd98f43d1cd4acc88eed6cbc39e946dbe -> trunk/9ecd092bd98f43d1cd4acc88eed6cbc39e946dbe 2025-10-10T01:23:15.2142961Z * [new tag] trunk/9f5e1beaf3c9248a335d2448103240a463187eb5 -> trunk/9f5e1beaf3c9248a335d2448103240a463187eb5 2025-10-10T01:23:15.2144474Z * [new tag] trunk/9fc2c6446d394dd313ed71e9d1ffc4f7f3916423 -> trunk/9fc2c6446d394dd313ed71e9d1ffc4f7f3916423 2025-10-10T01:23:15.2145934Z * [new tag] trunk/9fff8155c362da777e7ce31b85fb2dc7cfced2d5 -> trunk/9fff8155c362da777e7ce31b85fb2dc7cfced2d5 2025-10-10T01:23:15.2147394Z * [new tag] trunk/a029675f6f0b9cf48eb7943d4be8169c67960a8e -> trunk/a029675f6f0b9cf48eb7943d4be8169c67960a8e 2025-10-10T01:23:15.2148823Z * [new tag] trunk/a11a66ef320938cd0fd72b44b2b572b06937e100 -> trunk/a11a66ef320938cd0fd72b44b2b572b06937e100 2025-10-10T01:23:15.2150402Z * [new tag] trunk/a2f29bcd6388acdc3202d8a90974c50ffb605104 -> trunk/a2f29bcd6388acdc3202d8a90974c50ffb605104 2025-10-10T01:23:15.2151922Z * [new tag] trunk/a34797e031727f6a01a2f13a66db2f7e1fcc05b6 -> trunk/a34797e031727f6a01a2f13a66db2f7e1fcc05b6 2025-10-10T01:23:15.2153474Z * [new tag] trunk/a4110fedcf72eaede76324bb5c21a76589d75849 -> trunk/a4110fedcf72eaede76324bb5c21a76589d75849 2025-10-10T01:23:15.2154675Z * [new tag] trunk/a43c4c3972a611db169dde2aed803b91fe78c081 -> trunk/a43c4c3972a611db169dde2aed803b91fe78c081 2025-10-10T01:23:15.2156135Z * [new tag] trunk/a57a14868dcfd9dabf9bd19b6b11f31967c80c87 -> trunk/a57a14868dcfd9dabf9bd19b6b11f31967c80c87 2025-10-10T01:23:15.2157697Z * [new tag] trunk/a6fa4f9c283971c0fb6f60a89674a1f35370ac79 -> trunk/a6fa4f9c283971c0fb6f60a89674a1f35370ac79 2025-10-10T01:23:15.2159189Z * [new tag] trunk/a753ffa9aff47e005c31d6bcbf5b6a61cc54afed -> trunk/a753ffa9aff47e005c31d6bcbf5b6a61cc54afed 2025-10-10T01:23:15.2160708Z * [new tag] trunk/a7fa1a91e386c7708e4c8747680911b0c3174a66 -> trunk/a7fa1a91e386c7708e4c8747680911b0c3174a66 2025-10-10T01:23:15.2161984Z * [new tag] trunk/a9a9a3438a374f96a308b707a1718036aaec790d -> trunk/a9a9a3438a374f96a308b707a1718036aaec790d 2025-10-10T01:23:15.2163872Z * [new tag] trunk/ab01a0d7d352e7fd07989b8d6bf035bf82aea74e -> trunk/ab01a0d7d352e7fd07989b8d6bf035bf82aea74e 2025-10-10T01:23:15.2165319Z * [new tag] trunk/ab94a0d544503b5c27e889b45e45ef8cf75c8183 -> trunk/ab94a0d544503b5c27e889b45e45ef8cf75c8183 2025-10-10T01:23:15.2166840Z * [new tag] trunk/abadea70f3eb5f2f764fd6448d42dd2c29fa28b3 -> trunk/abadea70f3eb5f2f764fd6448d42dd2c29fa28b3 2025-10-10T01:23:15.2168177Z * [new tag] trunk/ac08556f674259ff5b117964e300124e8a92d45b -> trunk/ac08556f674259ff5b117964e300124e8a92d45b 2025-10-10T01:23:15.2169674Z * [new tag] trunk/ac7b4e7fe4d233dcd7f6343d42b4fa3d64bce548 -> trunk/ac7b4e7fe4d233dcd7f6343d42b4fa3d64bce548 2025-10-10T01:23:15.2171105Z * [new tag] trunk/ac901bf79a2d78539ffec272bf32f4ae47035b23 -> trunk/ac901bf79a2d78539ffec272bf32f4ae47035b23 2025-10-10T01:23:15.2172562Z * [new tag] trunk/ad7b2bebc651c297d869f265deedef726bf17048 -> trunk/ad7b2bebc651c297d869f265deedef726bf17048 2025-10-10T01:23:15.2174132Z * [new tag] trunk/ae25ec569c614c2a2274837079578b71f3201a3b -> trunk/ae25ec569c614c2a2274837079578b71f3201a3b 2025-10-10T01:23:15.2175887Z * [new tag] trunk/aea57b3aa38a3d4a058e0a7eba08d0c6c28ed9c5 -> trunk/aea57b3aa38a3d4a058e0a7eba08d0c6c28ed9c5 2025-10-10T01:23:15.2177361Z * [new tag] trunk/aed5ed1076d3e73e0b6357dafac1002aa6a221e9 -> trunk/aed5ed1076d3e73e0b6357dafac1002aa6a221e9 2025-10-10T01:23:15.2178823Z * [new tag] trunk/aed66248a01d309eb2ac1149b5f51310545b0783 -> trunk/aed66248a01d309eb2ac1149b5f51310545b0783 2025-10-10T01:23:15.2180330Z * [new tag] trunk/af32d16a71681ca05c6d410fb1b9cee091d4577d -> trunk/af32d16a71681ca05c6d410fb1b9cee091d4577d 2025-10-10T01:23:15.2181807Z * [new tag] trunk/af40828bbb785f968eda18dbdc8750ba67f57366 -> trunk/af40828bbb785f968eda18dbdc8750ba67f57366 2025-10-10T01:23:15.2183262Z * [new tag] trunk/af4c29fea8f50ac3bb9e4a0e305da4a2c6b53d29 -> trunk/af4c29fea8f50ac3bb9e4a0e305da4a2c6b53d29 2025-10-10T01:23:15.2184715Z * [new tag] trunk/afee8062d511ad63e0af65ffac0e712d86aae8f1 -> trunk/afee8062d511ad63e0af65ffac0e712d86aae8f1 2025-10-10T01:23:15.2186190Z * [new tag] trunk/afeec56a5aa83dd0258565400551a99777c0023b -> trunk/afeec56a5aa83dd0258565400551a99777c0023b 2025-10-10T01:23:15.2187788Z * [new tag] trunk/b0985144b59db8fb20964829b5e0a9d2f9a3f0d6 -> trunk/b0985144b59db8fb20964829b5e0a9d2f9a3f0d6 2025-10-10T01:23:15.2189393Z * [new tag] trunk/b116c5133024be39a2db67cd0112b490b970b710 -> trunk/b116c5133024be39a2db67cd0112b490b970b710 2025-10-10T01:23:15.2190873Z * [new tag] trunk/b13cd141b3585c2ae89ad7747acd11203a2fb837 -> trunk/b13cd141b3585c2ae89ad7747acd11203a2fb837 2025-10-10T01:23:15.2192395Z * [new tag] trunk/b1ac252f55f4a4d0e5488fb2ac9154154decec87 -> trunk/b1ac252f55f4a4d0e5488fb2ac9154154decec87 2025-10-10T01:23:15.2193954Z * [new tag] trunk/b28b24a9fc7d391c5793a94489a3f2d5381f6ad7 -> trunk/b28b24a9fc7d391c5793a94489a3f2d5381f6ad7 2025-10-10T01:23:15.2195302Z * [new tag] trunk/b2b3947565fd0c27ebd4941152c964eab30370e2 -> trunk/b2b3947565fd0c27ebd4941152c964eab30370e2 2025-10-10T01:23:15.2196863Z * [new tag] trunk/b558c986e8ec693b531ad2817026393c55d72eb6 -> trunk/b558c986e8ec693b531ad2817026393c55d72eb6 2025-10-10T01:23:15.2198370Z * [new tag] trunk/b5e93ffdcf779c703af5c8119636b01f250eafcd -> trunk/b5e93ffdcf779c703af5c8119636b01f250eafcd 2025-10-10T01:23:15.2199855Z * [new tag] trunk/b63bbe16615cc7680836dbb151bd848bce4893d6 -> trunk/b63bbe16615cc7680836dbb151bd848bce4893d6 2025-10-10T01:23:15.2201187Z * [new tag] trunk/b6b7a44dec63495d57946cbfe8f2accb8f876db2 -> trunk/b6b7a44dec63495d57946cbfe8f2accb8f876db2 2025-10-10T01:23:15.2202497Z * [new tag] trunk/b9e73e639e36f3aa628752161711e68878231b30 -> trunk/b9e73e639e36f3aa628752161711e68878231b30 2025-10-10T01:23:15.2204150Z * [new tag] trunk/ba480d6bf78ea446d1268d9b5b3a0dbb490c9c88 -> trunk/ba480d6bf78ea446d1268d9b5b3a0dbb490c9c88 2025-10-10T01:23:15.2205656Z * [new tag] trunk/bac0f289a35f05052740076fc5671271a3d487c2 -> trunk/bac0f289a35f05052740076fc5671271a3d487c2 2025-10-10T01:23:15.2207155Z * [new tag] trunk/bc1690c7e859dee8c47a7f0bbd3c43cc27c6fd2a -> trunk/bc1690c7e859dee8c47a7f0bbd3c43cc27c6fd2a 2025-10-10T01:23:15.2208494Z * [new tag] trunk/bc33b10202fb7c3761bcabc166e02d96807d8739 -> trunk/bc33b10202fb7c3761bcabc166e02d96807d8739 2025-10-10T01:23:15.2209932Z * [new tag] trunk/bcafea5c92ca2ee1b0dc8f6d8b62ecabb6f40228 -> trunk/bcafea5c92ca2ee1b0dc8f6d8b62ecabb6f40228 2025-10-10T01:23:15.2211341Z * [new tag] trunk/bcd96cc6ff798281e66aabef6ce72542fdc97c7a -> trunk/bcd96cc6ff798281e66aabef6ce72542fdc97c7a 2025-10-10T01:23:15.2212787Z * [new tag] trunk/bd3b98a8a5d68ddc84b20a4609b9ea90998bf95b -> trunk/bd3b98a8a5d68ddc84b20a4609b9ea90998bf95b 2025-10-10T01:23:15.2214287Z * [new tag] trunk/bdc0a421d7bcc49db12f7593d2c213a6141da614 -> trunk/bdc0a421d7bcc49db12f7593d2c213a6141da614 2025-10-10T01:23:15.2215810Z * [new tag] trunk/bde18c445dcb1d83e8ea0afae52f9b9bf8171f45 -> trunk/bde18c445dcb1d83e8ea0afae52f9b9bf8171f45 2025-10-10T01:23:15.2217143Z * [new tag] trunk/bf717ce346203fc27e792f4bdcc31e979cd74fa9 -> trunk/bf717ce346203fc27e792f4bdcc31e979cd74fa9 2025-10-10T01:23:15.2218835Z * [new tag] trunk/c0510dc447a1f105cb8758d2721380f7a7c380d1 -> trunk/c0510dc447a1f105cb8758d2721380f7a7c380d1 2025-10-10T01:23:15.2220292Z * [new tag] trunk/c1f40d33c89b361a1edad17aa25cfff1ab4014fd -> trunk/c1f40d33c89b361a1edad17aa25cfff1ab4014fd 2025-10-10T01:23:15.2221726Z * [new tag] trunk/c32118dc3e50505fd285e6e448a90883fce11535 -> trunk/c32118dc3e50505fd285e6e448a90883fce11535 2025-10-10T01:23:15.2223180Z * [new tag] trunk/c45d56dd00546daa7d9044674233dba1ac7b6194 -> trunk/c45d56dd00546daa7d9044674233dba1ac7b6194 2025-10-10T01:23:15.2224610Z * [new tag] trunk/c6329524d8670d5f9295cddcf7ebc3040ed9179e -> trunk/c6329524d8670d5f9295cddcf7ebc3040ed9179e 2025-10-10T01:23:15.2226071Z * [new tag] trunk/c6a6c80a730ff4edaec0d2fc4a5ff9344edaed41 -> trunk/c6a6c80a730ff4edaec0d2fc4a5ff9344edaed41 2025-10-10T01:23:15.2227354Z * [new tag] trunk/c7b57d93490d83c9cf59908eb34754882664a1c2 -> trunk/c7b57d93490d83c9cf59908eb34754882664a1c2 2025-10-10T01:23:15.2228893Z * [new tag] trunk/c7e30ae4dd9a58ed4f4bcbdc6afc2249cac94f28 -> trunk/c7e30ae4dd9a58ed4f4bcbdc6afc2249cac94f28 2025-10-10T01:23:15.2230359Z * [new tag] trunk/c813617c53e6be91e77f47e9a3f713146d54f340 -> trunk/c813617c53e6be91e77f47e9a3f713146d54f340 2025-10-10T01:23:15.2232004Z * [new tag] trunk/c855f8632e331b51d60d5f1bcc59d3181cb4bc82 -> trunk/c855f8632e331b51d60d5f1bcc59d3181cb4bc82 2025-10-10T01:23:15.2233366Z * [new tag] trunk/c965d6dbb23a8b2338ffebf3f01c6f92ce5847d2 -> trunk/c965d6dbb23a8b2338ffebf3f01c6f92ce5847d2 2025-10-10T01:23:15.2234906Z * [new tag] trunk/cac5e13e1384900c5acc4938c33d6037a61850d5 -> trunk/cac5e13e1384900c5acc4938c33d6037a61850d5 2025-10-10T01:23:15.2236373Z * [new tag] trunk/cc71ab86a6985e85645424b727c766e031047ff6 -> trunk/cc71ab86a6985e85645424b727c766e031047ff6 2025-10-10T01:23:15.2237838Z * [new tag] trunk/cd62a73dcb13102069aa827a6657f62d88cce095 -> trunk/cd62a73dcb13102069aa827a6657f62d88cce095 2025-10-10T01:23:15.2239349Z * [new tag] trunk/cf0a00d4f38775e5a82a166e367f40383c606963 -> trunk/cf0a00d4f38775e5a82a166e367f40383c606963 2025-10-10T01:23:15.2240909Z * [new tag] trunk/cfc5cc17dc4fa6be41b4b31eb6e63d3863479452 -> trunk/cfc5cc17dc4fa6be41b4b31eb6e63d3863479452 2025-10-10T01:23:15.2242364Z * [new tag] trunk/cfd46d13e6e1308add3a9f287b4855ccc3f2e66c -> trunk/cfd46d13e6e1308add3a9f287b4855ccc3f2e66c 2025-10-10T01:23:15.2243760Z * [new tag] trunk/d1a62c80363cf769552453eed187e935f905737d -> trunk/d1a62c80363cf769552453eed187e935f905737d 2025-10-10T01:23:15.2245199Z * [new tag] trunk/d1cbb74fb16406488a174832e1b58b7c242f418d -> trunk/d1cbb74fb16406488a174832e1b58b7c242f418d 2025-10-10T01:23:15.2246647Z * [new tag] trunk/d386325ca9a142419f45b987391f4bb175dd7d0b -> trunk/d386325ca9a142419f45b987391f4bb175dd7d0b 2025-10-10T01:23:15.2248107Z * [new tag] trunk/d40a9bfb8da0dc1ac1e6e56b33a25979112874de -> trunk/d40a9bfb8da0dc1ac1e6e56b33a25979112874de 2025-10-10T01:23:15.2249578Z * [new tag] trunk/d4443840036a00a30afcf066cb23f4525e590809 -> trunk/d4443840036a00a30afcf066cb23f4525e590809 2025-10-10T01:23:15.2251050Z * [new tag] trunk/d4752bc7f6818a3df5356a9de61afe1d3e27ade9 -> trunk/d4752bc7f6818a3df5356a9de61afe1d3e27ade9 2025-10-10T01:23:15.2252515Z * [new tag] trunk/da49a57d3462332b26cb7ee58910b5bc67e5772c -> trunk/da49a57d3462332b26cb7ee58910b5bc67e5772c 2025-10-10T01:23:15.2253983Z * [new tag] trunk/da903b6a8be422529d47649e89c0d50bb95c37ca -> trunk/da903b6a8be422529d47649e89c0d50bb95c37ca 2025-10-10T01:23:15.2255500Z * [new tag] trunk/dca73982c53e9f99f96246b5d9ed9bab83c7423f -> trunk/dca73982c53e9f99f96246b5d9ed9bab83c7423f 2025-10-10T01:23:15.2256980Z * [new tag] trunk/ddf8de28c25944a58e739ba9996b06753e4199cc -> trunk/ddf8de28c25944a58e739ba9996b06753e4199cc 2025-10-10T01:23:15.2258828Z * [new tag] trunk/df640df68a5275684eaae3080a9c97a0c61469c8 -> trunk/df640df68a5275684eaae3080a9c97a0c61469c8 2025-10-10T01:23:15.2260224Z * [new tag] trunk/e09fb44ef177005c4a11c28be24781429d416a3e -> trunk/e09fb44ef177005c4a11c28be24781429d416a3e 2025-10-10T01:23:15.2261734Z * [new tag] trunk/e0cb1848d0fd9fb4467ad8b844c565aea5071838 -> trunk/e0cb1848d0fd9fb4467ad8b844c565aea5071838 2025-10-10T01:23:15.2263296Z * [new tag] trunk/e3ae80fc036da356e3748d134689741583552f09 -> trunk/e3ae80fc036da356e3748d134689741583552f09 2025-10-10T01:23:15.2264881Z * [new tag] trunk/e40fe634b1a7aa33e278b1404ee02dea12277080 -> trunk/e40fe634b1a7aa33e278b1404ee02dea12277080 2025-10-10T01:23:15.2266348Z * [new tag] trunk/e438db254602cf39ba536aed0590b4144c019ee8 -> trunk/e438db254602cf39ba536aed0590b4144c019ee8 2025-10-10T01:23:15.2267830Z * [new tag] trunk/e532f62e0d96e56cb28fa6a0ba6d981896a65d52 -> trunk/e532f62e0d96e56cb28fa6a0ba6d981896a65d52 2025-10-10T01:23:15.2269252Z * [new tag] trunk/e6d4b26776842307475b368db60e27ac1bcede86 -> trunk/e6d4b26776842307475b368db60e27ac1bcede86 2025-10-10T01:23:15.2270710Z * [new tag] trunk/e7ed1a00eb5510d1c7dccd17b5c0ebb54231284f -> trunk/e7ed1a00eb5510d1c7dccd17b5c0ebb54231284f 2025-10-10T01:23:15.2272263Z * [new tag] trunk/e7fd2969303ab931f5e5875eca676018e1acd089 -> trunk/e7fd2969303ab931f5e5875eca676018e1acd089 2025-10-10T01:23:15.2273676Z * [new tag] trunk/e89d12bf5d6b69c153cd000ef278fca59f03226d -> trunk/e89d12bf5d6b69c153cd000ef278fca59f03226d 2025-10-10T01:23:15.2275118Z * [new tag] trunk/e98c4e835b1db22092fc93b49d2cddd7b3537d1f -> trunk/e98c4e835b1db22092fc93b49d2cddd7b3537d1f 2025-10-10T01:23:15.2276520Z * [new tag] trunk/ea42517e454d2e47391646bbb897f5fc51147b9d -> trunk/ea42517e454d2e47391646bbb897f5fc51147b9d 2025-10-10T01:23:15.2277954Z * [new tag] trunk/eaa02655eabd24609744c2251ac40d39d86ebb39 -> trunk/eaa02655eabd24609744c2251ac40d39d86ebb39 2025-10-10T01:23:15.2279502Z * [new tag] trunk/eccf561326147894d57482a5aba7a2290005b257 -> trunk/eccf561326147894d57482a5aba7a2290005b257 2025-10-10T01:23:15.2281153Z * [new tag] trunk/ece5e0f01b68509342f85fa388ca61936dc18b20 -> trunk/ece5e0f01b68509342f85fa388ca61936dc18b20 2025-10-10T01:23:15.2282526Z * [new tag] trunk/ed2d514ad860229f6d364688f9db27dad034cd83 -> trunk/ed2d514ad860229f6d364688f9db27dad034cd83 2025-10-10T01:23:15.2283960Z * [new tag] trunk/ed6156e3ea334b9b8d395e5a9f76fa3ba7408c06 -> trunk/ed6156e3ea334b9b8d395e5a9f76fa3ba7408c06 2025-10-10T01:23:15.2285394Z * [new tag] trunk/ee5389d520844db36374e86c986b9ff8f47ac4bb -> trunk/ee5389d520844db36374e86c986b9ff8f47ac4bb 2025-10-10T01:23:15.2286831Z * [new tag] trunk/ee6a1ecb0a1035f068484c8fcfba44b2efc9e837 -> trunk/ee6a1ecb0a1035f068484c8fcfba44b2efc9e837 2025-10-10T01:23:15.2288270Z * [new tag] trunk/ef50c6e3e3d83bfd67e50930eea9a3a9db084061 -> trunk/ef50c6e3e3d83bfd67e50930eea9a3a9db084061 2025-10-10T01:23:15.2289737Z * [new tag] trunk/ef7e2ca77e3f554ced81eb614f15fb84249d4a7e -> trunk/ef7e2ca77e3f554ced81eb614f15fb84249d4a7e 2025-10-10T01:23:15.2291236Z * [new tag] trunk/f006aee601cb72077f4b1dbc3f7f0f685e57a1a9 -> trunk/f006aee601cb72077f4b1dbc3f7f0f685e57a1a9 2025-10-10T01:23:15.2292707Z * [new tag] trunk/f05e23e1bc1439e19145e43e8ffca0051cda2f33 -> trunk/f05e23e1bc1439e19145e43e8ffca0051cda2f33 2025-10-10T01:23:15.2294144Z * [new tag] trunk/f0c9f3bddbf7ad77d5d3a8803c23bb47bfb71d79 -> trunk/f0c9f3bddbf7ad77d5d3a8803c23bb47bfb71d79 2025-10-10T01:23:15.2295605Z * [new tag] trunk/f11ac803d73b90d7e1f7bde962b9afe6b5967eb7 -> trunk/f11ac803d73b90d7e1f7bde962b9afe6b5967eb7 2025-10-10T01:23:15.2297746Z * [new tag] trunk/f1229b6db946c290cc5bdea05dde69fc01e0bed0 -> trunk/f1229b6db946c290cc5bdea05dde69fc01e0bed0 2025-10-10T01:23:15.2299247Z * [new tag] trunk/f231be25c679adb47ac3e483dc68948e5ad137a4 -> trunk/f231be25c679adb47ac3e483dc68948e5ad137a4 2025-10-10T01:23:15.2300794Z * [new tag] trunk/f33201729416ed17467228e80b04d01d4d02b5f3 -> trunk/f33201729416ed17467228e80b04d01d4d02b5f3 2025-10-10T01:23:15.2302261Z * [new tag] trunk/f37a6523efe1b9bf7f6b5b5d0f36dc461a3fda2a -> trunk/f37a6523efe1b9bf7f6b5b5d0f36dc461a3fda2a 2025-10-10T01:23:15.2303781Z * [new tag] trunk/f39789cdabb6465f21666bd001829e1f7284d754 -> trunk/f39789cdabb6465f21666bd001829e1f7284d754 2025-10-10T01:23:15.2305223Z * [new tag] trunk/f3afbcf3407783e54ec2795b06ae744f645320ba -> trunk/f3afbcf3407783e54ec2795b06ae744f645320ba 2025-10-10T01:23:15.2306686Z * [new tag] trunk/f3e43ff2d73f375487b1b71483bbecb6cdad8920 -> trunk/f3e43ff2d73f375487b1b71483bbecb6cdad8920 2025-10-10T01:23:15.2308142Z * [new tag] trunk/f414aa8e0d17e8eff38a93cebd52436e53f50eba -> trunk/f414aa8e0d17e8eff38a93cebd52436e53f50eba 2025-10-10T01:23:15.2309580Z * [new tag] trunk/f465ea6752c91498de63eb57439a74f4836e568a -> trunk/f465ea6752c91498de63eb57439a74f4836e568a 2025-10-10T01:23:15.2311297Z * [new tag] trunk/f46bb04dcc37a9b394e414569aef8aef69f9bf53 -> trunk/f46bb04dcc37a9b394e414569aef8aef69f9bf53 2025-10-10T01:23:15.2312603Z * [new tag] trunk/f46ddb1e65b595c80f285dc42aa8549970736aae -> trunk/f46ddb1e65b595c80f285dc42aa8549970736aae 2025-10-10T01:23:15.2314005Z * [new tag] trunk/f4cf75688f0fd93466589addfb7d0ec33e46e3bf -> trunk/f4cf75688f0fd93466589addfb7d0ec33e46e3bf 2025-10-10T01:23:15.2315494Z * [new tag] trunk/f505caa71bd2e4d1e708e20a3665b834134e08fc -> trunk/f505caa71bd2e4d1e708e20a3665b834134e08fc 2025-10-10T01:23:15.2316946Z * [new tag] trunk/f5fd18f7e24378bd9eb91404f697f1c81a8187d5 -> trunk/f5fd18f7e24378bd9eb91404f697f1c81a8187d5 2025-10-10T01:23:15.2318365Z * [new tag] trunk/f6de195616432f42a545b98ea41cc816019d1c60 -> trunk/f6de195616432f42a545b98ea41cc816019d1c60 2025-10-10T01:23:15.2319954Z * [new tag] trunk/f6f76767563d4293a0f78551edf4675a5794c570 -> trunk/f6f76767563d4293a0f78551edf4675a5794c570 2025-10-10T01:23:15.2321397Z * [new tag] trunk/f7082e92b3635e89906fae514506152a2ec844a0 -> trunk/f7082e92b3635e89906fae514506152a2ec844a0 2025-10-10T01:23:15.2322900Z * [new tag] trunk/f713abab16cb98c15f486e9822dd261279cce252 -> trunk/f713abab16cb98c15f486e9822dd261279cce252 2025-10-10T01:23:15.2324385Z * [new tag] trunk/f76fdcaaf8b6d5f97c7f63705400ebed8984f869 -> trunk/f76fdcaaf8b6d5f97c7f63705400ebed8984f869 2025-10-10T01:23:15.2325868Z * [new tag] trunk/f79e212733ca89ce3cc99a3072e50351686e5568 -> trunk/f79e212733ca89ce3cc99a3072e50351686e5568 2025-10-10T01:23:15.2327368Z * [new tag] trunk/f7ad6dbad67161333a1473d1e0b478b7475a0ec1 -> trunk/f7ad6dbad67161333a1473d1e0b478b7475a0ec1 2025-10-10T01:23:15.2328799Z * [new tag] trunk/fa5306b4f5bea89d80b9f14926712119aab78161 -> trunk/fa5306b4f5bea89d80b9f14926712119aab78161 2025-10-10T01:23:15.2330268Z * [new tag] trunk/fac6f20ae3a68fa49e19571a1fc4bcdddbf87d80 -> trunk/fac6f20ae3a68fa49e19571a1fc4bcdddbf87d80 2025-10-10T01:23:15.2331743Z * [new tag] trunk/fac85fcfb5ad0e63438d808a2f9ba7ea2dff9ad4 -> trunk/fac85fcfb5ad0e63438d808a2f9ba7ea2dff9ad4 2025-10-10T01:23:15.2333166Z * [new tag] trunk/fd3e15c14f4fc474af610b482382a2c85729f50d -> trunk/fd3e15c14f4fc474af610b482382a2c85729f50d 2025-10-10T01:23:15.2334686Z * [new tag] trunk/fd4bde430a51e5f216295c950d962c6343119821 -> trunk/fd4bde430a51e5f216295c950d962c6343119821 2025-10-10T01:23:15.2336169Z * [new tag] trunk/fdc622b513610b53ddcdc0b40282df9beae369bd -> trunk/fdc622b513610b53ddcdc0b40282df9beae369bd 2025-10-10T01:23:15.2337633Z * [new tag] trunk/fdc8ccc5bc433478c2a114016e193f5665d1e370 -> trunk/fdc8ccc5bc433478c2a114016e193f5665d1e370 2025-10-10T01:23:15.2338995Z * [new tag] trunk/ff5faa744a52561f4c6a138089123fd8d41cab73 -> trunk/ff5faa744a52561f4c6a138089123fd8d41cab73 2025-10-10T01:23:15.2340303Z * [new tag] v0.1.1 -> v0.1.1 2025-10-10T01:23:15.2341660Z * [new tag] v0.1.10 -> v0.1.10 2025-10-10T01:23:15.2342903Z * [new tag] v0.1.11 -> v0.1.11 2025-10-10T01:23:15.2344280Z * [new tag] v0.1.12 -> v0.1.12 2025-10-10T01:23:15.2345558Z * [new tag] v0.1.2 -> v0.1.2 2025-10-10T01:23:15.2346856Z * [new tag] v0.1.3 -> v0.1.3 2025-10-10T01:23:15.2348189Z * [new tag] v0.1.4 -> v0.1.4 2025-10-10T01:23:15.2349522Z * [new tag] v0.1.5 -> v0.1.5 2025-10-10T01:23:15.2350900Z * [new tag] v0.1.6 -> v0.1.6 2025-10-10T01:23:15.2352140Z * [new tag] v0.1.7 -> v0.1.7 2025-10-10T01:23:15.2353501Z * [new tag] v0.1.8 -> v0.1.8 2025-10-10T01:23:15.2354869Z * [new tag] v0.1.9 -> v0.1.9 2025-10-10T01:23:15.2356255Z * [new tag] v0.2.0 -> v0.2.0 2025-10-10T01:23:15.2357615Z * [new tag] v0.3.0 -> v0.3.0 2025-10-10T01:23:15.2359026Z * [new tag] v0.3.1 -> v0.3.1 2025-10-10T01:23:15.2360523Z * [new tag] v0.4.0 -> v0.4.0 2025-10-10T01:23:15.2361826Z * [new tag] v0.4.1 -> v0.4.1 2025-10-10T01:23:15.2363161Z * [new tag] v1.0.0 -> v1.0.0 2025-10-10T01:23:15.2364509Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-10-10T01:23:15.2365825Z * [new tag] v1.0.1 -> v1.0.1 2025-10-10T01:23:15.2367383Z * [new tag] v1.0rc0 -> v1.0rc0 2025-10-10T01:23:15.2368552Z * [new tag] v1.0rc1 -> v1.0rc1 2025-10-10T01:23:15.2369956Z * [new tag] v1.1.0 -> v1.1.0 2025-10-10T01:23:15.2371351Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-10-10T01:23:15.2372975Z * [new tag] v1.10.0 -> v1.10.0 2025-10-10T01:23:15.2374352Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-10-10T01:23:15.2375776Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-10-10T01:23:15.2376938Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-10-10T01:23:15.2378337Z * [new tag] v1.10.1 -> v1.10.1 2025-10-10T01:23:15.2379506Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-10-10T01:23:15.2380741Z * [new tag] v1.10.2 -> v1.10.2 2025-10-10T01:23:15.2382343Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-10-10T01:23:15.2383769Z * [new tag] v1.11.0 -> v1.11.0 2025-10-10T01:23:15.2385116Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-10-10T01:23:15.2386595Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-10-10T01:23:15.2388001Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-10-10T01:23:15.2389394Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-10-10T01:23:15.2390834Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-10-10T01:23:15.2392004Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-10-10T01:23:15.2393184Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-10-10T01:23:15.2394509Z * [new tag] v1.12.0 -> v1.12.0 2025-10-10T01:23:15.2395840Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-10-10T01:23:15.2397487Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-10-10T01:23:15.2398931Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-10-10T01:23:15.2400537Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-10-10T01:23:15.2401852Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-10-10T01:23:15.2403335Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-10-10T01:23:15.2404486Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-10-10T01:23:15.2405662Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-10-10T01:23:15.2406836Z * [new tag] v1.12.1 -> v1.12.1 2025-10-10T01:23:15.2408315Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-10-10T01:23:15.2409767Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-10-10T01:23:15.2411356Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-10-10T01:23:15.2412675Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-10-10T01:23:15.2413849Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-10-10T01:23:15.2415243Z * [new tag] v1.13.0 -> v1.13.0 2025-10-10T01:23:15.2416682Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-10-10T01:23:15.2417991Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-10-10T01:23:15.2419316Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-10-10T01:23:15.2420797Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-10-10T01:23:15.2421927Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-10-10T01:23:15.2423106Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-10-10T01:23:15.2424590Z * [new tag] v1.13.1 -> v1.13.1 2025-10-10T01:23:15.2425777Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-10-10T01:23:15.2427151Z * [new tag] v1.2.0 -> v1.2.0 2025-10-10T01:23:15.2428564Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-10-10T01:23:15.2429915Z * [new tag] v1.3.0 -> v1.3.0 2025-10-10T01:23:15.2431299Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-10-10T01:23:15.2432529Z * [new tag] v1.3.1 -> v1.3.1 2025-10-10T01:23:15.2433999Z * [new tag] v1.4.0 -> v1.4.0 2025-10-10T01:23:15.2435324Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-10-10T01:23:15.2436498Z * [new tag] v1.4.1 -> v1.4.1 2025-10-10T01:23:15.2437938Z * [new tag] v1.5.0 -> v1.5.0 2025-10-10T01:23:15.2439465Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-10-10T01:23:15.2441136Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-10-10T01:23:15.2442510Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-10-10T01:23:15.2443834Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-10-10T01:23:15.2445039Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-10-10T01:23:15.2446487Z * [new tag] v1.5.1 -> v1.5.1 2025-10-10T01:23:15.2447652Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-10-10T01:23:15.2448826Z * [new tag] v1.6.0 -> v1.6.0 2025-10-10T01:23:15.2450259Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-10-10T01:23:15.2451659Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-10-10T01:23:15.2453078Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-10-10T01:23:15.2454453Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-10-10T01:23:15.2455834Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-10-10T01:23:15.2457215Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-10-10T01:23:15.2458381Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-10-10T01:23:15.2459815Z * [new tag] v1.7.0 -> v1.7.0 2025-10-10T01:23:15.2461175Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-10-10T01:23:15.2462644Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-10-10T01:23:15.2463998Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-10-10T01:23:15.2465165Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-10-10T01:23:15.2466802Z * [new tag] v1.7.1 -> v1.7.1 2025-10-10T01:23:15.2468140Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-10-10T01:23:15.2469859Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-10-10T01:23:15.2471059Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-10-10T01:23:15.2472470Z * [new tag] v1.8.0 -> v1.8.0 2025-10-10T01:23:15.2473658Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-10-10T01:23:15.2475030Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-10-10T01:23:15.2476465Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-10-10T01:23:15.2477836Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-10-10T01:23:15.2479099Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-10-10T01:23:15.2480411Z * [new tag] v1.8.1 -> v1.8.1 2025-10-10T01:23:15.2481803Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-10-10T01:23:15.2482973Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-10-10T01:23:15.2484181Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-10-10T01:23:15.2485931Z * [new tag] v1.8.2 -> v1.8.2 2025-10-10T01:23:15.2487169Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-10-10T01:23:15.2488629Z * [new tag] v1.9.0 -> v1.9.0 2025-10-10T01:23:15.2489955Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-10-10T01:23:15.2491460Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-10-10T01:23:15.2492886Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-10-10T01:23:15.2494095Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-10-10T01:23:15.2495477Z * [new tag] v1.9.1 -> v1.9.1 2025-10-10T01:23:15.2497359Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-10-10T01:23:15.2498497Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-10-10T01:23:15.2499904Z * [new tag] v2.0.0 -> v2.0.0 2025-10-10T01:23:15.2501242Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-10-10T01:23:15.2502737Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-10-10T01:23:15.2504119Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-10-10T01:23:15.2505417Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-10-10T01:23:15.2506833Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-10-10T01:23:15.2507988Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-10-10T01:23:15.2509453Z * [new tag] v2.0.1 -> v2.0.1 2025-10-10T01:23:15.2510871Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-10-10T01:23:15.2511992Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-10-10T01:23:15.2513302Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-10-10T01:23:15.2514486Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-10-10T01:23:15.2516280Z * [new tag] v2.1.0 -> v2.1.0 2025-10-10T01:23:15.2517600Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-10-10T01:23:15.2519016Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-10-10T01:23:15.2520563Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-10-10T01:23:15.2521973Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-10-10T01:23:15.2523453Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-10-10T01:23:15.2524767Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-10-10T01:23:15.2526047Z * [new tag] v2.1.1 -> v2.1.1 2025-10-10T01:23:15.2527404Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-10-10T01:23:15.2528802Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-10-10T01:23:15.2530278Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-10-10T01:23:15.2531829Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-10-10T01:23:15.2533163Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-10-10T01:23:15.2534361Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-10-10T01:23:15.2535775Z * [new tag] v2.1.2 -> v2.1.2 2025-10-10T01:23:15.2537282Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-10-10T01:23:15.2538727Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-10-10T01:23:15.2539912Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-10-10T01:23:15.2541360Z * [new tag] v2.2.0 -> v2.2.0 2025-10-10T01:23:15.2542730Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-10-10T01:23:15.2544025Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-10-10T01:23:15.2545346Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-10-10T01:23:15.2546747Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-10-10T01:23:15.2548164Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-10-10T01:23:15.2549627Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-10-10T01:23:15.2550778Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-10-10T01:23:15.2551959Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-10-10T01:23:15.2553354Z * [new tag] v2.2.1 -> v2.2.1 2025-10-10T01:23:15.2554806Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-10-10T01:23:15.2556091Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-10-10T01:23:15.2557652Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-10-10T01:23:15.2558858Z * [new tag] v2.2.2 -> v2.2.2 2025-10-10T01:23:15.2560468Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-10-10T01:23:15.2561617Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-10-10T01:23:15.2562804Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-10-10T01:23:15.2564189Z * [new tag] v2.3.0 -> v2.3.0 2025-10-10T01:23:15.2565650Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-10-10T01:23:15.2567137Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-10-10T01:23:15.2568554Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-10-10T01:23:15.2569733Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-10-10T01:23:15.2571154Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-10-10T01:23:15.2572653Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-10-10T01:23:15.2574037Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-10-10T01:23:15.2575470Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-10-10T01:23:15.2576652Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-10-10T01:23:15.2578026Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-10-10T01:23:15.2579370Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-10-10T01:23:15.2580682Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-10-10T01:23:15.2581806Z * [new tag] v2.3.1 -> v2.3.1 2025-10-10T01:23:15.2583126Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-10-10T01:23:15.2584468Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-10-10T01:23:15.2585818Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-10-10T01:23:15.2587238Z * [new tag] v2.4.0 -> v2.4.0 2025-10-10T01:23:15.2588613Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-10-10T01:23:15.2589945Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-10-10T01:23:15.2591309Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-10-10T01:23:15.2592694Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-10-10T01:23:15.2594122Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-10-10T01:23:15.2595508Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-10-10T01:23:15.2597974Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-10-10T01:23:15.2599501Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-10-10T01:23:15.2600903Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-10-10T01:23:15.2602011Z * [new tag] v2.4.1 -> v2.4.1 2025-10-10T01:23:15.2603465Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-10-10T01:23:15.2604804Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-10-10T01:23:15.2606228Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-10-10T01:23:15.2607645Z * [new tag] v2.5.0 -> v2.5.0 2025-10-10T01:23:15.2609014Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-10-10T01:23:15.2610262Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-10-10T01:23:15.2611743Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-10-10T01:23:15.2613537Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-10-10T01:23:15.2615008Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-10-10T01:23:15.2616385Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-10-10T01:23:15.2617830Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-10-10T01:23:15.2619189Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-10-10T01:23:15.2620680Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-10-10T01:23:15.2622020Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-10-10T01:23:15.2623211Z * [new tag] v2.5.1 -> v2.5.1 2025-10-10T01:23:15.2624407Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-10-10T01:23:15.2625607Z * [new tag] v2.6.0 -> v2.6.0 2025-10-10T01:23:15.2627043Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-10-10T01:23:15.2628516Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-10-10T01:23:15.2629885Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-10-10T01:23:15.2631385Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-10-10T01:23:15.2633160Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-10-10T01:23:15.2634666Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-10-10T01:23:15.2636022Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-10-10T01:23:15.2637521Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-10-10T01:23:15.2639090Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-10-10T01:23:15.2640627Z * [new tag] v2.7.0 -> v2.7.0 2025-10-10T01:23:15.2641911Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-10-10T01:23:15.2643118Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-10-10T01:23:15.2644538Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-10-10T01:23:15.2646044Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-10-10T01:23:15.2647831Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-10-10T01:23:15.2649186Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-10-10T01:23:15.2650564Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-10-10T01:23:15.2651995Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-10-10T01:23:15.2653464Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-10-10T01:23:15.2654880Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-10-10T01:23:15.2656043Z * [new tag] v2.7.1 -> v2.7.1 2025-10-10T01:23:15.2657550Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-10-10T01:23:15.2658912Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-10-10T01:23:15.2660420Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-10-10T01:23:15.2661867Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-10-10T01:23:15.2663336Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-10-10T01:23:15.2664572Z * [new tag] v2.8.0 -> v2.8.0 2025-10-10T01:23:15.2666065Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-10-10T01:23:15.2667494Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-10-10T01:23:15.2668962Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-10-10T01:23:15.2670368Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-10-10T01:23:15.2671820Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-10-10T01:23:15.2673257Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-10-10T01:23:15.2674742Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-10-10T01:23:15.2676187Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-10-10T01:23:15.2677574Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-10-10T01:23:15.2679290Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-10-10T01:23:15.2680798Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-10-10T01:23:15.2682159Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-10-10T01:23:15.2683486Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-10-10T01:23:15.2685062Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-10-10T01:23:15.2686368Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-10-10T01:23:15.2688006Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-10-10T01:23:15.2689246Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-10-10T01:23:15.2691119Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-10-10T01:23:15.2692580Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-10-10T01:23:15.2693741Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-10-10T01:23:15.2695100Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-10-10T01:23:15.2696655Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-10-10T01:23:15.2698045Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-10-10T01:23:15.2699335Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-10-10T01:23:15.2700518Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-10-10T01:23:15.2701820Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-10-10T01:23:15.2703471Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-10-10T01:23:15.2705082Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-10-10T01:23:15.2706494Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-10-10T01:23:15.2707947Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-10-10T01:23:15.2709278Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-10-10T01:23:15.2710520Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-10-10T01:23:15.2711822Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-10-10T01:23:15.2713237Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-10-10T01:23:15.2714355Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-10-10T01:23:15.2715616Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-10-10T01:23:15.2716854Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-10-10T01:23:15.2718149Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-10-10T01:23:15.2719419Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-10-10T01:23:15.2720744Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-10-10T01:23:15.2721943Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-10-10T01:23:15.2723228Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-10-10T01:23:15.2724645Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-10-10T01:23:15.2725884Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-10-10T01:23:15.2727126Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-10-10T01:23:15.2728514Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-10-10T01:23:15.2729751Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-10-10T01:23:15.2731018Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-10-10T01:23:15.2732293Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-10-10T01:23:15.2733637Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-10-10T01:23:15.2734826Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-10-10T01:23:15.2736070Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-10-10T01:23:15.2737335Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-10-10T01:23:15.2738615Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-10-10T01:23:15.2739842Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-10-10T01:23:15.2741067Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-10-10T01:23:15.2742316Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-10-10T01:23:15.2743665Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-10-10T01:23:15.2744933Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-10-10T01:23:15.2746157Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-10-10T01:23:15.2747414Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-10-10T01:23:15.2748665Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-10-10T01:23:15.2749977Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-10-10T01:23:15.2751260Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-10-10T01:23:15.2752527Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-10-10T01:23:15.2753805Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-10-10T01:23:15.2755057Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-10-10T01:23:15.2756299Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-10-10T01:23:15.2757516Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-10-10T01:23:15.2758820Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-10-10T01:23:15.2760133Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-10-10T01:23:15.2761373Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-10-10T01:23:15.2762561Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-10-10T01:23:15.2763830Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-10-10T01:23:15.2765090Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-10-10T01:23:15.2766363Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-10-10T01:23:15.2767608Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-10-10T01:23:15.2768862Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-10-10T01:23:15.2770586Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-10-10T01:23:15.2771939Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-10-10T01:23:15.2773228Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-10-10T01:23:15.2774514Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-10-10T01:23:15.2775836Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-10-10T01:23:15.2777127Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-10-10T01:23:15.2778338Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-10-10T01:23:15.2779715Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-10-10T01:23:15.2780831Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-10-10T01:23:15.2782048Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-10-10T01:23:15.2783299Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-10-10T01:23:15.2784555Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-10-10T01:23:15.2785784Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-10-10T01:23:15.2787054Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-10-10T01:23:15.2788406Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-10-10T01:23:15.2789726Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-10-10T01:23:15.2790976Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-10-10T01:23:15.2792180Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-10-10T01:23:15.2793453Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-10-10T01:23:15.2794701Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-10-10T01:23:15.2795946Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-10-10T01:23:15.2797475Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-10-10T01:23:15.2798743Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-10-10T01:23:15.2800049Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-10-10T01:23:15.2801307Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-10-10T01:23:15.2802542Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-10-10T01:23:15.2803826Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-10-10T01:23:15.2805137Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-10-10T01:23:15.2806329Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-10-10T01:23:15.2807568Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-10-10T01:23:15.2808906Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-10-10T01:23:15.2810124Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-10-10T01:23:15.2811384Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-10-10T01:23:15.2812684Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-10-10T01:23:15.2813961Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-10-10T01:23:15.2815198Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-10-10T01:23:15.2816455Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-10-10T01:23:15.2817732Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-10-10T01:23:15.2818998Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-10-10T01:23:15.2820217Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-10-10T01:23:15.2821480Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-10-10T01:23:15.2822885Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-10-10T01:23:15.2824175Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-10-10T01:23:15.2825420Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-10-10T01:23:15.2826698Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-10-10T01:23:15.2827947Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-10-10T01:23:15.2829234Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-10-10T01:23:15.2830475Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-10-10T01:23:15.2831810Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-10-10T01:23:15.2833038Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-10-10T01:23:15.2834313Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-10-10T01:23:15.2835705Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-10-10T01:23:15.2836898Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-10-10T01:23:15.2838141Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-10-10T01:23:15.2839490Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-10-10T01:23:15.2840888Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-10-10T01:23:15.2842224Z * [new tag] whc_flight_1 -> whc_flight_1 2025-10-10T01:23:15.2843660Z * [new tag] whc_flight_2 -> whc_flight_2 2025-10-10T01:23:15.2845296Z * [new tag] whc_flight_4 -> whc_flight_4 2025-10-10T01:23:15.3948346Z [command]/usr/bin/git rev-parse --verify --quiet 344e6365a0068c2d2847fcec0c55dd53291d475e^{object} 2025-10-10T01:23:15.3975053Z 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:23:15.3979338Z ##[endgroup] 2025-10-10T01:23:15.3979601Z ##[group]Determining the checkout info 2025-10-10T01:23:15.3981205Z ##[endgroup] 2025-10-10T01:23:15.3984588Z [command]/usr/bin/git sparse-checkout disable 2025-10-10T01:23:15.4021739Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-10-10T01:23:15.4048882Z ##[group]Checking out the ref 2025-10-10T01:23:15.4052522Z [command]/usr/bin/git checkout --progress --force 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:23:16.4314144Z Updating files: 73% (14695/19929) 2025-10-10T01:23:16.4580675Z Updating files: 74% (14748/19929) 2025-10-10T01:23:16.4970411Z Updating files: 75% (14947/19929) 2025-10-10T01:23:16.5201858Z Updating files: 76% (15147/19929) 2025-10-10T01:23:16.5351709Z Updating files: 77% (15346/19929) 2025-10-10T01:23:16.5513456Z Updating files: 78% (15545/19929) 2025-10-10T01:23:16.5805724Z Updating files: 79% (15744/19929) 2025-10-10T01:23:16.6078934Z Updating files: 80% (15944/19929) 2025-10-10T01:23:16.6334803Z Updating files: 81% (16143/19929) 2025-10-10T01:23:16.6588840Z Updating files: 82% (16342/19929) 2025-10-10T01:23:16.6768646Z Updating files: 83% (16542/19929) 2025-10-10T01:23:16.6922333Z Updating files: 84% (16741/19929) 2025-10-10T01:23:16.7101757Z Updating files: 85% (16940/19929) 2025-10-10T01:23:16.7277793Z Updating files: 86% (17139/19929) 2025-10-10T01:23:16.7438521Z Updating files: 87% (17339/19929) 2025-10-10T01:23:16.7585675Z Updating files: 88% (17538/19929) 2025-10-10T01:23:16.7740471Z Updating files: 89% (17737/19929) 2025-10-10T01:23:16.7931916Z Updating files: 90% (17937/19929) 2025-10-10T01:23:16.8088427Z Updating files: 91% (18136/19929) 2025-10-10T01:23:16.8250451Z Updating files: 92% (18335/19929) 2025-10-10T01:23:16.8462006Z Updating files: 93% (18534/19929) 2025-10-10T01:23:16.8678966Z Updating files: 94% (18734/19929) 2025-10-10T01:23:16.8872356Z Updating files: 95% (18933/19929) 2025-10-10T01:23:16.9049398Z Updating files: 96% (19132/19929) 2025-10-10T01:23:16.9232486Z Updating files: 97% (19332/19929) 2025-10-10T01:23:16.9507284Z Updating files: 98% (19531/19929) 2025-10-10T01:23:16.9695960Z Updating files: 99% (19730/19929) 2025-10-10T01:23:16.9696423Z Updating files: 100% (19929/19929) 2025-10-10T01:23:16.9696724Z Updating files: 100% (19929/19929), done. 2025-10-10T01:23:16.9929090Z Note: switching to '344e6365a0068c2d2847fcec0c55dd53291d475e'. 2025-10-10T01:23:16.9929396Z 2025-10-10T01:23:16.9929600Z You are in 'detached HEAD' state. You can look around, make experimental 2025-10-10T01:23:16.9930077Z changes and commit them, and you can discard any commits you make in this 2025-10-10T01:23:16.9930554Z state without impacting any branches by switching back to a branch. 2025-10-10T01:23:16.9930844Z 2025-10-10T01:23:16.9931025Z If you want to create a new branch to retain commits you create, you may 2025-10-10T01:23:16.9931464Z do so (now or later) by using -c with the switch command. Example: 2025-10-10T01:23:16.9931720Z 2025-10-10T01:23:16.9932107Z git switch -c 2025-10-10T01:23:16.9932312Z 2025-10-10T01:23:16.9932417Z Or undo this operation with: 2025-10-10T01:23:16.9932593Z 2025-10-10T01:23:16.9932685Z git switch - 2025-10-10T01:23:16.9932799Z 2025-10-10T01:23:16.9932970Z Turn off this advice by setting config variable advice.detachedHead to false 2025-10-10T01:23:16.9933216Z 2025-10-10T01:23:16.9935325Z HEAD is now at 344e6365a00 [inductor][eazy] change how torch.use_deterministic_algorithms affect inductor (#164905) 2025-10-10T01:23:17.0059921Z ##[endgroup] 2025-10-10T01:23:17.0060308Z ##[group]Setting up auth for fetching submodules 2025-10-10T01:23:17.0066090Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-10-10T01:23:17.0133736Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-10-10T01:23:17.0163774Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-10-10T01:23:17.0195364Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-10-10T01:23:17.0228117Z ##[endgroup] 2025-10-10T01:23:17.0228505Z ##[group]Fetching submodules 2025-10-10T01:23:17.0231982Z [command]/usr/bin/git submodule sync --recursive 2025-10-10T01:23:17.0616141Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-10-10T01:23:17.0981755Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-10-10T01:23:17.0985045Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-10-10T01:23:17.0989007Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-10-10T01:23:17.0993026Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-10-10T01:23:17.0997422Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-10-10T01:23:17.1002131Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-10-10T01:23:17.1006114Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-10-10T01:23:17.1010431Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-10-10T01:23:17.1014946Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-10-10T01:23:17.1019662Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-10-10T01:23:17.1024351Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-10-10T01:23:17.1028963Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-10-10T01:23:17.1033854Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-10-10T01:23:17.1038666Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-10-10T01:23:17.1043670Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-10-10T01:23:17.1049433Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-10-10T01:23:17.1057215Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-10-10T01:23:17.1062501Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-10-10T01:23:17.1067744Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:23:17.1072900Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-10-10T01:23:17.1078440Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-10-10T01:23:17.1084335Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-10-10T01:23:17.1089822Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-10-10T01:23:17.1095233Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-10-10T01:23:17.1101792Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-10-10T01:23:17.1107645Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-10-10T01:23:17.1113457Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-10-10T01:23:17.1119498Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-10-10T01:23:17.1125581Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-10-10T01:23:17.1131527Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-10-10T01:23:17.1137828Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-10-10T01:23:17.1144131Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-10-10T01:23:17.1150547Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-10-10T01:23:17.1159647Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-10-10T01:23:17.1166353Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-10-10T01:23:17.1172925Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-10-10T01:23:17.1179641Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-10-10T01:23:17.1218189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-10-10T01:23:17.3727207Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-10-10T01:23:17.3727908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-10-10T01:23:17.3760683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-10-10T01:23:20.0652949Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-10-10T01:23:20.0653964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-10-10T01:23:20.0655072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-10-10T01:23:20.0656083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-10-10T01:23:20.0657030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-10-10T01:23:20.0657893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-10-10T01:23:20.0659143Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-10-10T01:23:20.0660173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-10-10T01:23:20.0661108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-10-10T01:23:20.0661968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-10-10T01:23:20.0662844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-10-10T01:23:20.0663749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-10-10T01:23:20.0664708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-10-10T01:23:20.0665631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-10-10T01:23:20.0666547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-10-10T01:23:20.0667450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-10-10T01:23:20.0668446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-10-10T01:23:20.0708834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-10-10T01:23:20.6326091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-10-10T01:23:20.6327566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-10-10T01:23:20.6328792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-10-10T01:23:20.6813271Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-10-10T01:23:22.8484327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-10-10T01:23:22.8485692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-10-10T01:23:22.8486829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-10-10T01:23:22.8487999Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-10-10T01:23:22.8527579Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-10-10T01:23:22.9486527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-10-10T01:23:35.3918063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-10-10T01:23:35.3918794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-10-10T01:23:35.3919488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-10-10T01:23:35.3920163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-10-10T01:23:35.3920807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-10-10T01:23:35.4119785Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-10-10T01:23:35.4283940Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-10-10T01:23:35.4412126Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-10-10T01:23:35.4732291Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-10-10T01:23:35.5609702Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-10-10T01:23:35.6156834Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-10-10T01:23:36.5426487Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-10-10T01:23:36.7200930Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-10-10T01:23:36.7226834Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:23:36.7260906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-10-10T01:23:40.2639625Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-10-10T01:23:40.2937386Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-10-10T01:23:40.7139030Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-10-10T01:23:40.7655981Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-10-10T01:23:40.8693823Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-10-10T01:23:40.9183522Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-10-10T01:23:41.6476172Z Submodule path 'third_party/cutlass': checked out 'f3fde58372d33e9a5650ba7b80fc48b3b49d40c8' 2025-10-10T01:23:41.8317994Z Submodule path 'third_party/fbgemm': checked out '3cefe0564a8c3de514a152d40a2b4770f2ee5be0' 2025-10-10T01:23:41.8344242Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-10-10T01:23:41.8348333Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:23:41.8352349Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:23:41.8356481Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-10-10T01:23:41.8360851Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-10-10T01:23:41.8365241Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:23:41.8369659Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-10-10T01:23:41.8403377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-10-10T01:23:43.1107231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-10-10T01:23:43.1108094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-10-10T01:23:43.1108886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-10-10T01:23:43.2107551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-10-10T01:23:45.1731562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-10-10T01:23:45.2731991Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-10-10T01:23:48.1865295Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-10-10T01:23:48.6084592Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-10-10T01:23:48.7152251Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-10-10T01:23:49.4477471Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '311f3c8e51dc0eb56310cfc6980bf63d0fbd7917' 2025-10-10T01:23:49.4991380Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T01:23:49.5145708Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-10-10T01:23:49.6404329Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-10-10T01:23:49.7270945Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-10-10T01:23:49.7295792Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:23:49.7299597Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:23:49.7333967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-10-10T01:23:52.9092513Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-10-10T01:23:53.2167386Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-10-10T01:23:53.8730160Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-10-10T01:23:54.0426104Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-10-10T01:23:54.0766514Z Submodule path 'third_party/fmt': checked out 'e424e3f2e607da02742f73db84873b8084fc714c' 2025-10-10T01:23:54.1203801Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-10-10T01:23:54.1534352Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-10-10T01:23:54.2047776Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T01:23:54.2202103Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-10-10T01:23:54.2223112Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-10-10T01:23:54.2256264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-10-10T01:24:07.5290150Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-10-10T01:24:07.5550680Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-10-10T01:24:07.6431595Z Submodule path 'third_party/kineto': checked out '001ba8eb519438592f79dbc8e86a349f5f6c6829' 2025-10-10T01:24:07.6455650Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:07.6459044Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:07.6462759Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:07.6498041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-10-10T01:24:08.2820997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-10-10T01:24:08.7586528Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-10-10T01:24:08.8536004Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-10-10T01:24:08.8557942Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:08.8561633Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:08.8565404Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:08.8569385Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:08.8573308Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:08.8577529Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:08.8581646Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:08.8586148Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:08.8590565Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:08.8626195Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-10-10T01:24:10.2563852Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-10-10T01:24:10.2565013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-10-10T01:24:10.2566041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-10-10T01:24:10.2567017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-10-10T01:24:10.2567968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-10-10T01:24:10.2568951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-10-10T01:24:10.2569923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-10-10T01:24:10.3564102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-10-10T01:24:16.1839439Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-10-10T01:24:16.2078335Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-10-10T01:24:16.2492099Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-10-10T01:24:16.2666030Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-10-10T01:24:16.2688134Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:16.2723209Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-10-10T01:24:16.5550873Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-10-10T01:24:16.5789930Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-10-10T01:24:16.6292000Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T01:24:16.7419520Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-10-10T01:24:16.7632677Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-10-10T01:24:16.7858132Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-10-10T01:24:16.7879989Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:16.7884066Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:16.7918425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-10-10T01:24:18.5375994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-10-10T01:24:18.7932746Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-10-10T01:24:18.8442614Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-10-10T01:24:18.8801955Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-10-10T01:24:18.9298099Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T01:24:18.9815413Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-10-10T01:24:19.0269745Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-10-10T01:24:19.1578712Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-10-10T01:24:19.7420319Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-10-10T01:24:19.7459592Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:19.7494777Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-10-10T01:24:20.7043510Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-10-10T01:24:20.7966488Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-10-10T01:24:20.7989766Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:20.7993520Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:20.7998025Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:20.8001884Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:20.8006424Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:20.8010353Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:20.8014454Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:20.8018503Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:20.8055328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-10-10T01:24:21.2395418Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-10-10T01:24:21.2397414Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-10-10T01:24:21.2398742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-10-10T01:24:21.2400145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-10-10T01:24:21.3397160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-10-10T01:24:21.8533370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-10-10T01:24:28.1418362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-10-10T01:24:28.5921856Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-10-10T01:24:28.6377492Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-10-10T01:24:28.6581290Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-10-10T01:24:28.7798933Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-10-10T01:24:28.7973802Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-10-10T01:24:28.8178566Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-10-10T01:24:28.8399396Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-10-10T01:24:28.8421117Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:28.8424744Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:28.8458644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-10-10T01:24:30.5820836Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-10-10T01:24:30.8326301Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-10-10T01:24:30.8847843Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-10-10T01:24:31.5899438Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-10-10T01:24:31.6055648Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-10-10T01:24:31.9120189Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-10-10T01:24:31.9148945Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:31.9152860Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:31.9187679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-10-10T01:24:32.4379689Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-10-10T01:24:32.8984756Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-10-10T01:24:32.9731012Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-10-10T01:24:32.9866298Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-10-10T01:24:33.0022683Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-10-10T01:24:33.0493878Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-10-10T01:24:33.0830919Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-10-10T01:24:33.1317799Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-10-10T01:24:33.1669458Z Submodule path 'third_party/tensorpipe': checked out 'af0118d13e52f5a08841464a768e01a0bf3e3075' 2025-10-10T01:24:33.1692649Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:33.1696493Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:33.1700358Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:33.1704168Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:33.1737219Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-10-10T01:24:34.2454848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-10-10T01:24:34.2507532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-10-10T01:24:34.3318661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-10-10T01:24:34.3920040Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-10-10T01:24:34.4122266Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-10-10T01:24:34.4922374Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-10-10T01:24:34.5275771Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-10-10T01:24:34.5296880Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:34.5329947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-10-10T01:24:34.7411927Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-10-10T01:24:34.7462226Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-10-10T01:24:34.7826206Z Entering 'android/libs/fbjni' 2025-10-10T01:24:34.7880761Z Entering 'third_party/FP16' 2025-10-10T01:24:34.7933887Z Entering 'third_party/FXdiv' 2025-10-10T01:24:34.7987671Z Entering 'third_party/NNPACK' 2025-10-10T01:24:34.8042091Z Entering 'third_party/NVTX' 2025-10-10T01:24:34.8098650Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:24:34.8152144Z Entering 'third_party/XNNPACK' 2025-10-10T01:24:34.8220372Z Entering 'third_party/aiter' 2025-10-10T01:24:34.8278392Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:24:34.8344513Z Entering 'third_party/benchmark' 2025-10-10T01:24:34.8397078Z Entering 'third_party/composable_kernel' 2025-10-10T01:24:34.8460671Z Entering 'third_party/cpp-httplib' 2025-10-10T01:24:34.8519389Z Entering 'third_party/cpuinfo' 2025-10-10T01:24:34.8575773Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:24:34.8632004Z Entering 'third_party/cutlass' 2025-10-10T01:24:34.8694114Z Entering 'third_party/fbgemm' 2025-10-10T01:24:34.8748743Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:24:34.8802711Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:24:34.8865017Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:24:34.8917924Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:24:34.8979951Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:24:34.9032600Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:24:34.9085000Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:24:34.9141426Z Entering 'third_party/flash-attention' 2025-10-10T01:24:34.9195785Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:24:34.9254165Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:24:34.9316550Z Entering 'third_party/flatbuffers' 2025-10-10T01:24:34.9379275Z Entering 'third_party/fmt' 2025-10-10T01:24:34.9433344Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:24:34.9488105Z Entering 'third_party/gloo' 2025-10-10T01:24:34.9540704Z Entering 'third_party/googletest' 2025-10-10T01:24:34.9596410Z Entering 'third_party/ideep' 2025-10-10T01:24:34.9651362Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:24:34.9714457Z Entering 'third_party/ittapi' 2025-10-10T01:24:34.9768421Z Entering 'third_party/kineto' 2025-10-10T01:24:34.9823843Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:34.9876436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:34.9933314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:34.9989390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:35.0043332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:35.0096908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:35.0150599Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:35.0207851Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:35.0260085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:35.0318513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:35.0371900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:35.0424415Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:35.0479897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:35.0539436Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:35.0593008Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:35.0650164Z Entering 'third_party/kleidiai' 2025-10-10T01:24:35.0705333Z Entering 'third_party/mimalloc' 2025-10-10T01:24:35.0761501Z Entering 'third_party/nlohmann' 2025-10-10T01:24:35.0817314Z Entering 'third_party/onnx' 2025-10-10T01:24:35.0886780Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:35.0943383Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:24:35.1001990Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:35.1055272Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:35.1109076Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:35.1160177Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:35.1216513Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:35.1267715Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:35.1320228Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:35.1371894Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:35.1425518Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:35.1480617Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:35.1552354Z Entering 'third_party/pocketfft' 2025-10-10T01:24:35.1606825Z Entering 'third_party/protobuf' 2025-10-10T01:24:35.1664024Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:35.1717810Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:35.1774525Z Entering 'third_party/psimd' 2025-10-10T01:24:35.1831479Z Entering 'third_party/pthreadpool' 2025-10-10T01:24:35.1886229Z Entering 'third_party/pybind11' 2025-10-10T01:24:35.1943126Z Entering 'third_party/python-peachpy' 2025-10-10T01:24:35.2001566Z Entering 'third_party/sleef' 2025-10-10T01:24:35.2057121Z Entering 'third_party/tensorpipe' 2025-10-10T01:24:35.2110587Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:35.2164795Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:35.2217659Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:35.2268510Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:35.2317808Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:35.2393925Z ##[endgroup] 2025-10-10T01:24:35.2394585Z ##[group]Persisting credentials for submodules 2025-10-10T01:24:35.2406794Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-10-10T01:24:35.2768401Z Entering 'android/libs/fbjni' 2025-10-10T01:24:35.2838393Z Entering 'third_party/FP16' 2025-10-10T01:24:35.2909286Z Entering 'third_party/FXdiv' 2025-10-10T01:24:35.2978411Z Entering 'third_party/NNPACK' 2025-10-10T01:24:35.3048505Z Entering 'third_party/NVTX' 2025-10-10T01:24:35.3119757Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:24:35.3188467Z Entering 'third_party/XNNPACK' 2025-10-10T01:24:35.3271067Z Entering 'third_party/aiter' 2025-10-10T01:24:35.3339089Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:24:35.3418000Z Entering 'third_party/benchmark' 2025-10-10T01:24:35.3489621Z Entering 'third_party/composable_kernel' 2025-10-10T01:24:35.3567594Z Entering 'third_party/cpp-httplib' 2025-10-10T01:24:35.3642173Z Entering 'third_party/cpuinfo' 2025-10-10T01:24:35.3715542Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:24:35.3787161Z Entering 'third_party/cutlass' 2025-10-10T01:24:35.3864470Z Entering 'third_party/fbgemm' 2025-10-10T01:24:35.3937744Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:24:35.4008073Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:24:35.4083948Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:24:35.4156851Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:24:35.4235168Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:24:35.4305208Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:24:35.4377954Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:24:35.4451461Z Entering 'third_party/flash-attention' 2025-10-10T01:24:35.4526515Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:24:35.4604732Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:24:35.4686715Z Entering 'third_party/flatbuffers' 2025-10-10T01:24:35.4760115Z Entering 'third_party/fmt' 2025-10-10T01:24:35.4829613Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:24:35.4901247Z Entering 'third_party/gloo' 2025-10-10T01:24:35.4975273Z Entering 'third_party/googletest' 2025-10-10T01:24:35.5048441Z Entering 'third_party/ideep' 2025-10-10T01:24:35.5117255Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:24:35.5194895Z Entering 'third_party/ittapi' 2025-10-10T01:24:35.5266901Z Entering 'third_party/kineto' 2025-10-10T01:24:35.5336298Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:35.5406418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:35.5478179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:35.5548161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:35.5619150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:35.5687205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:35.5761624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:35.5832222Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:35.5908726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:35.5979605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:35.6050343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:35.6118359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:35.6191322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:35.6267916Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:35.6338907Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:35.6411815Z Entering 'third_party/kleidiai' 2025-10-10T01:24:35.6485510Z Entering 'third_party/mimalloc' 2025-10-10T01:24:35.6557205Z Entering 'third_party/nlohmann' 2025-10-10T01:24:35.6628706Z Entering 'third_party/onnx' 2025-10-10T01:24:35.6713599Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:35.6788725Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:24:35.6859194Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:35.6928061Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:35.6998483Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:35.7067090Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:35.7139065Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:35.7209271Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:35.7278451Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:35.7346789Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:35.7419362Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:35.7491817Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:35.7583913Z Entering 'third_party/pocketfft' 2025-10-10T01:24:35.7657100Z Entering 'third_party/protobuf' 2025-10-10T01:24:35.7728659Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:35.7798748Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:35.7871055Z Entering 'third_party/psimd' 2025-10-10T01:24:35.7943442Z Entering 'third_party/pthreadpool' 2025-10-10T01:24:35.8018596Z Entering 'third_party/pybind11' 2025-10-10T01:24:35.8089905Z Entering 'third_party/python-peachpy' 2025-10-10T01:24:35.8160609Z Entering 'third_party/sleef' 2025-10-10T01:24:35.8229120Z Entering 'third_party/tensorpipe' 2025-10-10T01:24:35.8299139Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:35.8367922Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:35.8435820Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:35.8510052Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:35.8576224Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:35.8671196Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-10-10T01:24:35.9039065Z Entering 'android/libs/fbjni' 2025-10-10T01:24:35.9104402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-10-10T01:24:35.9126286Z Entering 'third_party/FP16' 2025-10-10T01:24:35.9190853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-10-10T01:24:35.9213456Z Entering 'third_party/FXdiv' 2025-10-10T01:24:35.9282594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-10-10T01:24:35.9304479Z Entering 'third_party/NNPACK' 2025-10-10T01:24:35.9370590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-10-10T01:24:35.9392795Z Entering 'third_party/NVTX' 2025-10-10T01:24:35.9462636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-10-10T01:24:35.9486033Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:24:35.9551760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-10-10T01:24:35.9574895Z Entering 'third_party/XNNPACK' 2025-10-10T01:24:35.9642057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-10-10T01:24:35.9677859Z Entering 'third_party/aiter' 2025-10-10T01:24:35.9740683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-10-10T01:24:35.9763139Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:24:35.9828550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-10-10T01:24:35.9858910Z Entering 'third_party/benchmark' 2025-10-10T01:24:35.9922447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-10-10T01:24:35.9944431Z Entering 'third_party/composable_kernel' 2025-10-10T01:24:36.0011447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-10-10T01:24:36.0041036Z Entering 'third_party/cpp-httplib' 2025-10-10T01:24:36.0106303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-10-10T01:24:36.0126985Z Entering 'third_party/cpuinfo' 2025-10-10T01:24:36.0192628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-10-10T01:24:36.0215846Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:24:36.0281920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-10-10T01:24:36.0303559Z Entering 'third_party/cutlass' 2025-10-10T01:24:36.0366559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-10-10T01:24:36.0395809Z Entering 'third_party/fbgemm' 2025-10-10T01:24:36.0462672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-10-10T01:24:36.0486176Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:24:36.0554326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-10-10T01:24:36.0575208Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:24:36.0642169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-10-10T01:24:36.0671966Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:24:36.0738542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-10-10T01:24:36.0759466Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:24:36.0824653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-10-10T01:24:36.0854786Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:24:36.0921092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-10-10T01:24:36.0941682Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:24:36.1006140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-10-10T01:24:36.1027021Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:24:36.1093493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-10-10T01:24:36.1119778Z Entering 'third_party/flash-attention' 2025-10-10T01:24:36.1184442Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-10-10T01:24:36.1206496Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:24:36.1271361Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-10-10T01:24:36.1300072Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:24:36.1364123Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-10-10T01:24:36.1397140Z Entering 'third_party/flatbuffers' 2025-10-10T01:24:36.1462527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-10-10T01:24:36.1488939Z Entering 'third_party/fmt' 2025-10-10T01:24:36.1553806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-10-10T01:24:36.1576865Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:24:36.1642956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-10-10T01:24:36.1665033Z Entering 'third_party/gloo' 2025-10-10T01:24:36.1731564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-10-10T01:24:36.1754269Z Entering 'third_party/googletest' 2025-10-10T01:24:36.1822109Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-10-10T01:24:36.1844627Z Entering 'third_party/ideep' 2025-10-10T01:24:36.1913628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-10-10T01:24:36.1934648Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:24:36.2001198Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-10-10T01:24:36.2031843Z Entering 'third_party/ittapi' 2025-10-10T01:24:36.2098597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-10-10T01:24:36.2119399Z Entering 'third_party/kineto' 2025-10-10T01:24:36.2183708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-10-10T01:24:36.2205025Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:36.2272607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-10-10T01:24:36.2294094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:36.2362653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-10-10T01:24:36.2386634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:36.2454607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-10-10T01:24:36.2476917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:36.2544807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-10-10T01:24:36.2567464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:36.2635689Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-10-10T01:24:36.2655744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:36.2724347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-10-10T01:24:36.2749937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:36.2816844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-10-10T01:24:36.2839281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:36.2905112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-10-10T01:24:36.2927668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:36.2994434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-10-10T01:24:36.3017978Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:36.3085236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-10-10T01:24:36.3110201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:36.3175754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-10-10T01:24:36.3196509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:36.3262470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-10-10T01:24:36.3287249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:36.3355517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-10-10T01:24:36.3384053Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:36.3450655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-10-10T01:24:36.3472707Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:36.3536057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-10-10T01:24:36.3560470Z Entering 'third_party/kleidiai' 2025-10-10T01:24:36.3629316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-10-10T01:24:36.3652775Z Entering 'third_party/mimalloc' 2025-10-10T01:24:36.3720248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-10-10T01:24:36.3740751Z Entering 'third_party/nlohmann' 2025-10-10T01:24:36.3804437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-10-10T01:24:36.3827623Z Entering 'third_party/onnx' 2025-10-10T01:24:36.3892983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-10-10T01:24:36.3929960Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:36.3994861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-10-10T01:24:36.4019772Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:24:36.4084456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-10-10T01:24:36.4106533Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:36.4172227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-10-10T01:24:36.4195973Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:36.4262135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-10-10T01:24:36.4284569Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:36.4354057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-10-10T01:24:36.4375692Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:36.4443934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-10-10T01:24:36.4464714Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:36.4532303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-10-10T01:24:36.4557043Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:36.4624497Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-10-10T01:24:36.4645810Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:36.4715001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-10-10T01:24:36.4734978Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:36.4803241Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-10-10T01:24:36.4827233Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:36.4893886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-10-10T01:24:36.4919192Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:36.4984479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-10-10T01:24:36.5025577Z Entering 'third_party/pocketfft' 2025-10-10T01:24:36.5092296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-10-10T01:24:36.5115499Z Entering 'third_party/protobuf' 2025-10-10T01:24:36.5180235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-10-10T01:24:36.5205899Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:36.5272683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-10-10T01:24:36.5294796Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:36.5362363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-10-10T01:24:36.5387990Z Entering 'third_party/psimd' 2025-10-10T01:24:36.5453430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-10-10T01:24:36.5477255Z Entering 'third_party/pthreadpool' 2025-10-10T01:24:36.5542151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-10-10T01:24:36.5565125Z Entering 'third_party/pybind11' 2025-10-10T01:24:36.5632928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-10-10T01:24:36.5656696Z Entering 'third_party/python-peachpy' 2025-10-10T01:24:36.5721586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-10-10T01:24:36.5744036Z Entering 'third_party/sleef' 2025-10-10T01:24:36.5812720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-10-10T01:24:36.5836968Z Entering 'third_party/tensorpipe' 2025-10-10T01:24:36.5901555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-10-10T01:24:36.5923290Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:36.5989920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-10-10T01:24:36.6011333Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:36.6075031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-10-10T01:24:36.6098196Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:36.6163247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-10-10T01:24:36.6185578Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:36.6253662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-10-10T01:24:36.6273862Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:36.6341614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-10-10T01:24:36.7052694Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-10-10T01:24:36.7419011Z Entering 'android/libs/fbjni' 2025-10-10T01:24:36.7474318Z Entering 'third_party/FP16' 2025-10-10T01:24:36.7530323Z Entering 'third_party/FXdiv' 2025-10-10T01:24:36.7582866Z Entering 'third_party/NNPACK' 2025-10-10T01:24:36.7638951Z Entering 'third_party/NVTX' 2025-10-10T01:24:36.7691368Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:24:36.7746116Z Entering 'third_party/XNNPACK' 2025-10-10T01:24:36.7811816Z Entering 'third_party/aiter' 2025-10-10T01:24:36.7864900Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:24:36.7927553Z Entering 'third_party/benchmark' 2025-10-10T01:24:36.7979697Z Entering 'third_party/composable_kernel' 2025-10-10T01:24:36.8042069Z Entering 'third_party/cpp-httplib' 2025-10-10T01:24:36.8095239Z Entering 'third_party/cpuinfo' 2025-10-10T01:24:36.8153584Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:24:36.8208545Z Entering 'third_party/cutlass' 2025-10-10T01:24:36.8269845Z Entering 'third_party/fbgemm' 2025-10-10T01:24:36.8326254Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:24:36.8379613Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:24:36.8441050Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:24:36.8494198Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:24:36.8556337Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:24:36.8609257Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:24:36.8661192Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:24:36.8721774Z Entering 'third_party/flash-attention' 2025-10-10T01:24:36.8775922Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:24:36.8838505Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:24:36.8900279Z Entering 'third_party/flatbuffers' 2025-10-10T01:24:36.8956582Z Entering 'third_party/fmt' 2025-10-10T01:24:36.9010468Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:24:36.9065753Z Entering 'third_party/gloo' 2025-10-10T01:24:36.9121005Z Entering 'third_party/googletest' 2025-10-10T01:24:36.9174547Z Entering 'third_party/ideep' 2025-10-10T01:24:36.9225801Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:24:36.9287888Z Entering 'third_party/ittapi' 2025-10-10T01:24:36.9341041Z Entering 'third_party/kineto' 2025-10-10T01:24:36.9394618Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:36.9447603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:36.9501906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:36.9554024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:36.9610357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:36.9662912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:36.9721309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:36.9774964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:36.9830813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:36.9887478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:36.9941212Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:36.9993408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:37.0050434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:37.0111973Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:37.0164994Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:37.0220137Z Entering 'third_party/kleidiai' 2025-10-10T01:24:37.0275560Z Entering 'third_party/mimalloc' 2025-10-10T01:24:37.0334325Z Entering 'third_party/nlohmann' 2025-10-10T01:24:37.0391274Z Entering 'third_party/onnx' 2025-10-10T01:24:37.0463530Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:37.0522434Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:24:37.0576602Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:37.0629243Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:37.0685002Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:37.0738561Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:37.0792183Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:37.0844754Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:37.0897456Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:37.0948916Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:37.1006154Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:37.1061830Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:37.1136402Z Entering 'third_party/pocketfft' 2025-10-10T01:24:37.1189592Z Entering 'third_party/protobuf' 2025-10-10T01:24:37.1246310Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:37.1299795Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:37.1356180Z Entering 'third_party/psimd' 2025-10-10T01:24:37.1411124Z Entering 'third_party/pthreadpool' 2025-10-10T01:24:37.1466630Z Entering 'third_party/pybind11' 2025-10-10T01:24:37.1520508Z Entering 'third_party/python-peachpy' 2025-10-10T01:24:37.1574413Z Entering 'third_party/sleef' 2025-10-10T01:24:37.1628613Z Entering 'third_party/tensorpipe' 2025-10-10T01:24:37.1683729Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:37.1738535Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:37.1790476Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:37.1844418Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:37.1896722Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:37.1971048Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-10-10T01:24:37.2332253Z Entering 'android/libs/fbjni' 2025-10-10T01:24:37.2385756Z Entering 'third_party/FP16' 2025-10-10T01:24:37.2440754Z Entering 'third_party/FXdiv' 2025-10-10T01:24:37.2493168Z Entering 'third_party/NNPACK' 2025-10-10T01:24:37.2548330Z Entering 'third_party/NVTX' 2025-10-10T01:24:37.2602553Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:24:37.2664273Z Entering 'third_party/XNNPACK' 2025-10-10T01:24:37.2731464Z Entering 'third_party/aiter' 2025-10-10T01:24:37.2784485Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:24:37.2846830Z Entering 'third_party/benchmark' 2025-10-10T01:24:37.2902740Z Entering 'third_party/composable_kernel' 2025-10-10T01:24:37.2965087Z Entering 'third_party/cpp-httplib' 2025-10-10T01:24:37.3019079Z Entering 'third_party/cpuinfo' 2025-10-10T01:24:37.3071220Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:24:37.3125524Z Entering 'third_party/cutlass' 2025-10-10T01:24:37.3188309Z Entering 'third_party/fbgemm' 2025-10-10T01:24:37.3244828Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:24:37.3299140Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:24:37.3358352Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:24:37.3411697Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:24:37.3471239Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:24:37.3524911Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:24:37.3577804Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:24:37.3634462Z Entering 'third_party/flash-attention' 2025-10-10T01:24:37.3689472Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:24:37.3747013Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:24:37.3809495Z Entering 'third_party/flatbuffers' 2025-10-10T01:24:37.3864892Z Entering 'third_party/fmt' 2025-10-10T01:24:37.3921041Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:24:37.3975041Z Entering 'third_party/gloo' 2025-10-10T01:24:37.4034042Z Entering 'third_party/googletest' 2025-10-10T01:24:37.4088281Z Entering 'third_party/ideep' 2025-10-10T01:24:37.4139063Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:24:37.4198342Z Entering 'third_party/ittapi' 2025-10-10T01:24:37.4253720Z Entering 'third_party/kineto' 2025-10-10T01:24:37.4307755Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:37.4359544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:37.4412726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:37.4472093Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:37.4527577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:37.4577792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:37.4637151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:37.4690949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:37.4745153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:37.4803528Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:37.4858427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:37.4910214Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:37.4968722Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:37.5029932Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:37.5085399Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:37.5143652Z Entering 'third_party/kleidiai' 2025-10-10T01:24:37.5199818Z Entering 'third_party/mimalloc' 2025-10-10T01:24:37.5255094Z Entering 'third_party/nlohmann' 2025-10-10T01:24:37.5311555Z Entering 'third_party/onnx' 2025-10-10T01:24:37.5384168Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:37.5443012Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:24:37.5497777Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:37.5550413Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:37.5605570Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:37.5657965Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:37.5711704Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:37.5764499Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:37.5818431Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:37.5869235Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:37.5923826Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:37.5978279Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:37.6049489Z Entering 'third_party/pocketfft' 2025-10-10T01:24:37.6103596Z Entering 'third_party/protobuf' 2025-10-10T01:24:37.6157557Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:37.6210672Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:37.6265814Z Entering 'third_party/psimd' 2025-10-10T01:24:37.6320068Z Entering 'third_party/pthreadpool' 2025-10-10T01:24:37.6374606Z Entering 'third_party/pybind11' 2025-10-10T01:24:37.6430049Z Entering 'third_party/python-peachpy' 2025-10-10T01:24:37.6481201Z Entering 'third_party/sleef' 2025-10-10T01:24:37.6535640Z Entering 'third_party/tensorpipe' 2025-10-10T01:24:37.6588774Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:37.6644801Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:37.6697209Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:37.6750067Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:37.6800273Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:37.6873716Z ##[endgroup] 2025-10-10T01:24:37.6913921Z [command]/usr/bin/git log -1 --format=%H 2025-10-10T01:24:37.6940138Z 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:24:37.7048260Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-10-10T01:24:37.7048527Z cd "${GITHUB_WORKSPACE}" 2025-10-10T01:24:37.7048749Z # Clean stale submodule dirs 2025-10-10T01:24:37.7048978Z if [ -z "${NO_SUDO}" ]; then 2025-10-10T01:24:37.7049256Z  sudo git submodule foreach --recursive git clean -ffdx 2025-10-10T01:24:37.7049531Z else 2025-10-10T01:24:37.7049766Z  git submodule foreach --recursive git clean -ffdx 2025-10-10T01:24:37.7050023Z fi 2025-10-10T01:24:37.7060761Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:37.7061054Z env: 2025-10-10T01:24:37.7061220Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:37.7061426Z NO_SUDO: true 2025-10-10T01:24:37.7061585Z ##[endgroup] 2025-10-10T01:24:37.7470272Z Entering 'android/libs/fbjni' 2025-10-10T01:24:37.7514525Z Entering 'third_party/FP16' 2025-10-10T01:24:37.7556562Z Entering 'third_party/FXdiv' 2025-10-10T01:24:37.7598240Z Entering 'third_party/NNPACK' 2025-10-10T01:24:37.7645978Z Entering 'third_party/NVTX' 2025-10-10T01:24:37.7700325Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:24:37.7744077Z Entering 'third_party/XNNPACK' 2025-10-10T01:24:37.7882000Z Entering 'third_party/aiter' 2025-10-10T01:24:37.7938910Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:24:37.8073466Z Entering 'third_party/benchmark' 2025-10-10T01:24:37.8117044Z Entering 'third_party/composable_kernel' 2025-10-10T01:24:37.8260686Z Entering 'third_party/cpp-httplib' 2025-10-10T01:24:37.8302835Z Entering 'third_party/cpuinfo' 2025-10-10T01:24:37.8349340Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:24:37.8398904Z Entering 'third_party/cutlass' 2025-10-10T01:24:37.8517643Z Entering 'third_party/fbgemm' 2025-10-10T01:24:37.8591690Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:24:37.8631239Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:24:37.8773697Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:24:37.8819524Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:24:37.8941328Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:24:37.8985885Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:24:37.9025839Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:24:37.9086031Z Entering 'third_party/flash-attention' 2025-10-10T01:24:37.9137148Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:24:37.9261015Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:24:37.9372086Z Entering 'third_party/flatbuffers' 2025-10-10T01:24:37.9467588Z Entering 'third_party/fmt' 2025-10-10T01:24:37.9510409Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:24:37.9553777Z Entering 'third_party/gloo' 2025-10-10T01:24:37.9597136Z Entering 'third_party/googletest' 2025-10-10T01:24:37.9641108Z Entering 'third_party/ideep' 2025-10-10T01:24:37.9679829Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:24:37.9783422Z Entering 'third_party/ittapi' 2025-10-10T01:24:37.9829144Z Entering 'third_party/kineto' 2025-10-10T01:24:37.9878579Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:24:37.9926333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:24:37.9983343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:24:38.0026566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:24:38.0071438Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:24:38.0109461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:24:38.0152569Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:24:38.0191431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:24:38.0235696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:24:38.0290640Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:24:38.0331179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:24:38.0373104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:38.0440600Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:38.0489701Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:24:38.0531589Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:24:38.0578026Z Entering 'third_party/kleidiai' 2025-10-10T01:24:38.0629040Z Entering 'third_party/mimalloc' 2025-10-10T01:24:38.0674069Z Entering 'third_party/nlohmann' 2025-10-10T01:24:38.0731384Z Entering 'third_party/onnx' 2025-10-10T01:24:38.1152833Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:24:38.1204703Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:24:38.1276909Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:24:38.1318810Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:24:38.1366157Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:24:38.1406439Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:24:38.1460748Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:24:38.1500942Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:24:38.1541353Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:24:38.1580709Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:24:38.1640793Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:24:38.1689141Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:24:38.2020190Z Entering 'third_party/pocketfft' 2025-10-10T01:24:38.2061024Z Entering 'third_party/protobuf' 2025-10-10T01:24:38.2159038Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:24:38.2199296Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:24:38.2245574Z Entering 'third_party/psimd' 2025-10-10T01:24:38.2286931Z Entering 'third_party/pthreadpool' 2025-10-10T01:24:38.2328574Z Entering 'third_party/pybind11' 2025-10-10T01:24:38.2374444Z Entering 'third_party/python-peachpy' 2025-10-10T01:24:38.2417904Z Entering 'third_party/sleef' 2025-10-10T01:24:38.2461312Z Entering 'third_party/tensorpipe' 2025-10-10T01:24:38.2506063Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:24:38.2548606Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:24:38.2589802Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:24:38.2635811Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:24:38.2678194Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:24:38.2833876Z Prepare all required actions 2025-10-10T01:24:38.2834384Z Getting action download info 2025-10-10T01:24:38.4162890Z ##[group]Run ./.github/actions/setup-linux 2025-10-10T01:24:38.4163129Z env: 2025-10-10T01:24:38.4163291Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:38.4163471Z ##[endgroup] 2025-10-10T01:24:38.4206427Z ##[group]Run set -euo pipefail 2025-10-10T01:24:38.4206685Z set -euo pipefail 2025-10-10T01:24:38.4206900Z function get_ec2_metadata() { 2025-10-10T01:24:38.4207170Z  # Pulled from instance metadata endpoint for EC2 2025-10-10T01:24:38.4207641Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-10-10T01:24:38.4208067Z  category=$1 2025-10-10T01:24:38.4208332Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-10-10T01:24:38.4208637Z  runner_name_str=i-0f89836b6c30bbee2 2025-10-10T01:24:38.4208903Z  if [[ -f /.inarc ]]; then 2025-10-10T01:24:38.4209163Z  echo "ARC Runner, no info on ec2 metadata" 2025-10-10T01:24:38.4209432Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-10-10T01:24:38.4209758Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-10-10T01:24:38.4210056Z  else 2025-10-10T01:24:38.4210651Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-10-10T01:24:38.4211268Z  fi 2025-10-10T01:24:38.4211430Z } 2025-10-10T01:24:38.4211624Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-10-10T01:24:38.4211921Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-10-10T01:24:38.4212250Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-10-10T01:24:38.4212537Z echo "system info $(uname -a)" 2025-10-10T01:24:38.4220776Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:38.4221070Z env: 2025-10-10T01:24:38.4221229Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:38.4221411Z ##[endgroup] 2025-10-10T01:24:38.4381715Z ami-id: ami-08982f1c5bf93d976 2025-10-10T01:24:38.4488231Z instance-id: i-0f89836b6c30bbee2 2025-10-10T01:24:38.4592590Z instance-type: g6.4xlarge 2025-10-10T01:24:38.4609371Z system info Linux ip-10-0-14-165.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-10-10T01:24:38.4635527Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-10-10T01:24:38.4635972Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-10-10T01:24:38.4643569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:38.4643839Z env: 2025-10-10T01:24:38.4643998Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:38.4644191Z ##[endgroup] 2025-10-10T01:24:39.8969185Z Fri Oct 10 01:24:39 2025 2025-10-10T01:24:39.8969554Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:24:39.8969934Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-10-10T01:24:39.8970306Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:24:39.8970679Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-10-10T01:24:39.8971074Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-10-10T01:24:39.8971688Z | | | MIG M. | 2025-10-10T01:24:39.8971942Z |=========================================+========================+======================| 2025-10-10T01:24:39.9047315Z | 0 NVIDIA L4 Off | 00000000:35:00.0 Off | 0 | 2025-10-10T01:24:39.9048005Z | N/A 49C P0 31W / 72W | 0MiB / 23034MiB | 4% Default | 2025-10-10T01:24:39.9048399Z | | | N/A | 2025-10-10T01:24:39.9048774Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:24:39.9049041Z 2025-10-10T01:24:39.9049200Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:24:39.9049589Z | Processes: | 2025-10-10T01:24:39.9050004Z | GPU GI CI PID Type Process name GPU Memory | 2025-10-10T01:24:39.9050386Z | ID ID Usage | 2025-10-10T01:24:39.9050692Z |=========================================================================================| 2025-10-10T01:24:39.9052264Z | No running processes found | 2025-10-10T01:24:39.9052720Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:24:40.2308035Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T01:24:40.2308709Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T01:24:40.2317636Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:40.2317945Z env: 2025-10-10T01:24:40.2318103Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:40.2318295Z ##[endgroup] 2025-10-10T01:24:40.2382807Z ##[group]Run if systemctl is-active --quiet docker; then 2025-10-10T01:24:40.2383125Z if systemctl is-active --quiet docker; then 2025-10-10T01:24:40.2383397Z  echo "Docker daemon is running..."; 2025-10-10T01:24:40.2383647Z else 2025-10-10T01:24:40.2383895Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-10-10T01:24:40.2384205Z fi 2025-10-10T01:24:40.2391944Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:40.2392223Z env: 2025-10-10T01:24:40.2392384Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:40.2392578Z ##[endgroup] 2025-10-10T01:24:40.2490484Z Docker daemon is running... 2025-10-10T01:24:40.2539667Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T01:24:40.2539891Z with: 2025-10-10T01:24:40.2540119Z shell: bash 2025-10-10T01:24:40.2540291Z timeout_minutes: 5 2025-10-10T01:24:40.2540497Z max_attempts: 3 2025-10-10T01:24:40.2540659Z retry_wait_seconds: 30 2025-10-10T01:24:40.2542324Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-10-10T01:24:40.2543929Z polling_interval_seconds: 1 2025-10-10T01:24:40.2544135Z warning_on_retry: true 2025-10-10T01:24:40.2544326Z continue_on_error: false 2025-10-10T01:24:40.2544507Z env: 2025-10-10T01:24:40.2544852Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:40.2545039Z AWS_RETRY_MODE: standard 2025-10-10T01:24:40.2545218Z AWS_MAX_ATTEMPTS: 5 2025-10-10T01:24:40.2545401Z AWS_DEFAULT_REGION: us-east-1 2025-10-10T01:24:40.2545589Z ##[endgroup] 2025-10-10T01:24:41.3151692Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T01:24:41.3152680Z Configure a credential helper to remove this warning. See 2025-10-10T01:24:41.3153295Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T01:24:41.3153660Z 2025-10-10T01:24:41.3153758Z Login Succeeded 2025-10-10T01:24:41.3278287Z Command completed after 1 attempt(s). 2025-10-10T01:24:41.3353249Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T01:24:41.3353623Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T01:24:41.3353944Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T01:24:41.3363821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:41.3364105Z env: 2025-10-10T01:24:41.3364267Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:41.3364461Z ##[endgroup] 2025-10-10T01:24:41.3464560Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T01:24:41.3464970Z # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T01:24:41.3465295Z # shellcheck disable=SC2046 2025-10-10T01:24:41.3465537Z docker stop $(docker ps -q) || true 2025-10-10T01:24:41.3465780Z # Prune all of the docker images 2025-10-10T01:24:41.3466012Z docker system prune -af 2025-10-10T01:24:41.3473553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:41.3473840Z env: 2025-10-10T01:24:41.3473994Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:41.3474182Z ##[endgroup] 2025-10-10T01:24:41.3764137Z "docker stop" requires at least 1 argument. 2025-10-10T01:24:41.3764654Z See 'docker stop --help'. 2025-10-10T01:24:41.3764857Z 2025-10-10T01:24:41.3765024Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-10-10T01:24:41.3765264Z 2025-10-10T01:24:41.3765367Z Stop one or more running containers 2025-10-10T01:24:41.4087695Z Total reclaimed space: 0B 2025-10-10T01:24:41.4244710Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-10-10T01:24:41.4245083Z with: 2025-10-10T01:24:41.4245687Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4246341Z use-custom-docker-registry: true 2025-10-10T01:24:41.4246568Z docker-build-dir: .ci/docker 2025-10-10T01:24:41.4246774Z docker-build-script: ./build.sh 2025-10-10T01:24:41.4246983Z working-directory: . 2025-10-10T01:24:41.4247231Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:41.4247511Z force-push: false 2025-10-10T01:24:41.4247693Z env: 2025-10-10T01:24:41.4247838Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:41.4248029Z ##[endgroup] 2025-10-10T01:24:41.4275891Z ##[group]Run set -ex 2025-10-10T01:24:41.4276110Z set -ex 2025-10-10T01:24:41.4276274Z  2025-10-10T01:24:41.4276581Z # If the docker build directory or the build script doesn't exist, the action will 2025-10-10T01:24:41.4277047Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-10-10T01:24:41.4277437Z # job could then download the pre-built image as usual 2025-10-10T01:24:41.4277915Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-10-10T01:24:41.4278360Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4278597Z else 2025-10-10T01:24:41.4278788Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4279221Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4279678Z  2025-10-10T01:24:41.4280075Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-10-10T01:24:41.4280513Z  exit 0 2025-10-10T01:24:41.4280664Z fi 2025-10-10T01:24:41.4280812Z  2025-10-10T01:24:41.4281050Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-10-10T01:24:41.4281465Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-10-10T01:24:41.4281833Z  # use it as it is, but first let's extract the tag 2025-10-10T01:24:41.4282164Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-10-10T01:24:41.4282529Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4282870Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4283148Z else 2025-10-10T01:24:41.4283335Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-10-10T01:24:41.4283593Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-10-10T01:24:41.4283866Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-10-10T01:24:41.4284097Z  fi 2025-10-10T01:24:41.4284416Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-10-10T01:24:41.4284829Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4285283Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4285775Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4286076Z fi 2025-10-10T01:24:41.4294408Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:41.4294692Z env: 2025-10-10T01:24:41.4294852Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:41.4295052Z REPO_NAME: pytorch 2025-10-10T01:24:41.4295779Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4296786Z DOCKER_BUILD_DIR: .ci/docker 2025-10-10T01:24:41.4297007Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-10-10T01:24:41.4297284Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:41.4297572Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-10-10T01:24:41.4297772Z CUSTOM_TAG_PREFIX: 2025-10-10T01:24:41.4297948Z ##[endgroup] 2025-10-10T01:24:41.4327193Z + [[ -d .ci/docker ]] 2025-10-10T01:24:41.4327496Z + [[ -f .ci/docker/./build.sh ]] 2025-10-10T01:24:41.4327765Z + [[ true == \t\r\u\e ]] 2025-10-10T01:24:41.4327995Z + echo skip=false 2025-10-10T01:24:41.4328914Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-10-10T01:24:41.4335291Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4335930Z ++ awk -F '[:,]' '{print $2}' 2025-10-10T01:24:41.4362424Z + DOCKER_TAG=pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4363293Z + echo docker-tag=pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4364375Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4397417Z ##[group]Run set +e 2025-10-10T01:24:41.4397793Z set +e 2025-10-10T01:24:41.4397963Z set -x 2025-10-10T01:24:41.4398117Z  2025-10-10T01:24:41.4398261Z login() { 2025-10-10T01:24:41.4398608Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-10-10T01:24:41.4399075Z } 2025-10-10T01:24:41.4399226Z  2025-10-10T01:24:41.4399369Z retry () { 2025-10-10T01:24:41.4399575Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-10-10T01:24:41.4399795Z } 2025-10-10T01:24:41.4399941Z  2025-10-10T01:24:41.4400097Z retry login "${DOCKER_REGISTRY}" 2025-10-10T01:24:41.4400311Z  2025-10-10T01:24:41.4400468Z START_TIME=$(date +%s) 2025-10-10T01:24:41.4400676Z # Wait up to 120 minutes 2025-10-10T01:24:41.4400933Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-10-10T01:24:41.4401277Z  # Check if image already exists, if it does then skip building it 2025-10-10T01:24:41.4401636Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-10-10T01:24:41.4401907Z  exit 0 2025-10-10T01:24:41.4402075Z  fi 2025-10-10T01:24:41.4402221Z  2025-10-10T01:24:41.4402507Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-10-10T01:24:41.4402969Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-10-10T01:24:41.4403427Z  # latter, it will wait for the Docker images to become available before continuing 2025-10-10T01:24:41.4403790Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-10-10T01:24:41.4404088Z  # It's a Docker build job, let's build the image 2025-10-10T01:24:41.4404341Z  break 2025-10-10T01:24:41.4404512Z  else 2025-10-10T01:24:41.4404756Z  # It's a regular build job, wait for the image to become available 2025-10-10T01:24:41.4405041Z  sleep 300 2025-10-10T01:24:41.4405222Z  fi 2025-10-10T01:24:41.4405376Z done 2025-10-10T01:24:41.4405522Z  2025-10-10T01:24:41.4405917Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-10-10T01:24:41.4406342Z # be empty. The default action would be to continue rebuild the image 2025-10-10T01:24:41.4406710Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-10-10T01:24:41.4407043Z  # if we're on the base branch then use the parent commit 2025-10-10T01:24:41.4407329Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-10-10T01:24:41.4407543Z else 2025-10-10T01:24:41.4407772Z  # otherwise we're on a PR, so use the most recent base commit 2025-10-10T01:24:41.4408107Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-10-10T01:24:41.4408363Z fi 2025-10-10T01:24:41.4408506Z  2025-10-10T01:24:41.4408676Z if [[ -z "${MERGE_BASE}" ]]; then 2025-10-10T01:24:41.4408924Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4409154Z  2025-10-10T01:24:41.4409475Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-10-10T01:24:41.4409865Z  exit 0 2025-10-10T01:24:41.4410033Z fi 2025-10-10T01:24:41.4410182Z  2025-10-10T01:24:41.4410398Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-10-10T01:24:41.4410867Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-10-10T01:24:41.4411270Z  exit 1 2025-10-10T01:24:41.4411427Z fi 2025-10-10T01:24:41.4411573Z  2025-10-10T01:24:41.4411819Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-10-10T01:24:41.4412276Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-10-10T01:24:41.4412777Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-10-10T01:24:41.4413257Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-10-10T01:24:41.4413789Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-10-10T01:24:41.4414100Z fi 2025-10-10T01:24:41.4414250Z  2025-10-10T01:24:41.4414431Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-10-10T01:24:41.4421395Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:41.4421670Z env: 2025-10-10T01:24:41.4421822Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:41.4422022Z DOCKER_BUILD_DIR: .ci/docker 2025-10-10T01:24:41.4422268Z BASE_REVISION: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:24:41.4422942Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4423791Z DOCKER_TAG: pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:41.4424301Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:41.4424584Z DOCKER_PUSH: 2025-10-10T01:24:41.4424759Z ##[endgroup] 2025-10-10T01:24:41.4452720Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:41.4453067Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:41.4456182Z + aws ecr get-login-password --region us-east-1 2025-10-10T01:24:41.4457052Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:41.9048127Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T01:24:41.9048711Z Configure a credential helper to remove this warning. See 2025-10-10T01:24:41.9049256Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T01:24:41.9049598Z 2025-10-10T01:24:41.9049752Z Login Succeeded 2025-10-10T01:24:41.9072314Z ++ date +%s 2025-10-10T01:24:41.9086057Z + START_TIME=1760059481 2025-10-10T01:24:41.9090138Z ++ date +%s 2025-10-10T01:24:41.9101200Z + [[ 1760052281 -lt 1760059481 ]] 2025-10-10T01:24:41.9102053Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:42.1485455Z { 2025-10-10T01:24:42.1485707Z "schemaVersion": 2, 2025-10-10T01:24:42.1486183Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-10-10T01:24:42.1486521Z "config": { 2025-10-10T01:24:42.1486779Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-10-10T01:24:42.1487079Z "size": 31341, 2025-10-10T01:24:42.1487379Z "digest": "sha256:fb0d053588b3e7d110638a9f678211e76aa4515ed1bad57389e38e2ab079e7b1" 2025-10-10T01:24:42.1487736Z }, 2025-10-10T01:24:42.1487874Z "layers": [ 2025-10-10T01:24:42.1488027Z { 2025-10-10T01:24:42.1488269Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1488570Z "size": 30447990, 2025-10-10T01:24:42.1488875Z "digest": "sha256:828c1365039a657352c737a62d13e1932951b5658eb6bd9b9096ea9b73562453" 2025-10-10T01:24:42.1489198Z }, 2025-10-10T01:24:42.1489340Z { 2025-10-10T01:24:42.1489571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1489855Z "size": 1554, 2025-10-10T01:24:42.1490372Z "digest": "sha256:140bb29742fd93f99a9a09900147573105606f902b116e98f74b52701d38c892" 2025-10-10T01:24:42.1490827Z }, 2025-10-10T01:24:42.1491058Z { 2025-10-10T01:24:42.1491381Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1491794Z "size": 313649951, 2025-10-10T01:24:42.1492290Z "digest": "sha256:99ee2db1caf9d1c2ddb56a54c2f9c956e2904d061a8f3c4cb990ec84d754fe89" 2025-10-10T01:24:42.1492924Z }, 2025-10-10T01:24:42.1504472Z { 2025-10-10T01:24:42.1504784Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1505106Z "size": 791, 2025-10-10T01:24:42.1505420Z "digest": "sha256:f5a2330f87a859f908a0c132821f161aca3279ad9972ba6d60cbf7f776ab7611" 2025-10-10T01:24:42.1505754Z }, 2025-10-10T01:24:42.1505949Z + exit 0 2025-10-10T01:24:42.1506093Z { 2025-10-10T01:24:42.1506333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1506645Z "size": 106, 2025-10-10T01:24:42.1506948Z "digest": "sha256:4643d3cb7e67efd8a0ab8abfd1953313811ea4d28f402ccfd94953bae4ee8df3" 2025-10-10T01:24:42.1507293Z }, 2025-10-10T01:24:42.1507492Z { 2025-10-10T01:24:42.1507746Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1508040Z "size": 703, 2025-10-10T01:24:42.1508320Z "digest": "sha256:5c871f60f6d4978a0a9d4b692a820cfa1b29c73e1244d641d570114156db0c50" 2025-10-10T01:24:42.1508646Z }, 2025-10-10T01:24:42.1508783Z { 2025-10-10T01:24:42.1509009Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1509286Z "size": 1214, 2025-10-10T01:24:42.1509568Z "digest": "sha256:60a2970be007b6e9760d42161c7b5ac43192a16730a7d13b7cd8f268757a952a" 2025-10-10T01:24:42.1509893Z }, 2025-10-10T01:24:42.1510030Z { 2025-10-10T01:24:42.1510244Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1510527Z "size": 485, 2025-10-10T01:24:42.1510806Z "digest": "sha256:43d427ee0b00452f669e2850ca0e4b80fbeb33cb3330ab2cbaab0e4fd27f4a49" 2025-10-10T01:24:42.1511127Z }, 2025-10-10T01:24:42.1511254Z { 2025-10-10T01:24:42.1511474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1511757Z "size": 110343651, 2025-10-10T01:24:42.1512053Z "digest": "sha256:728b6e1eb0128b7579525ddaa9ddbe96c7b04e6cb822fc419575c37c2bb3fbc4" 2025-10-10T01:24:42.1512425Z }, 2025-10-10T01:24:42.1512561Z { 2025-10-10T01:24:42.1512782Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1513059Z "size": 5038, 2025-10-10T01:24:42.1513564Z "digest": "sha256:1ffc73eae77b7a84ce198ea2a44254cb1ae80fbcc00e7b26877043256b7d8864" 2025-10-10T01:24:42.1513909Z }, 2025-10-10T01:24:42.1514040Z { 2025-10-10T01:24:42.1514263Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1514539Z "size": 1710, 2025-10-10T01:24:42.1514815Z "digest": "sha256:1c5379881a3b631c21f573017e01423112af82e3ac53f16992d23abb4d39aa27" 2025-10-10T01:24:42.1515129Z }, 2025-10-10T01:24:42.1515263Z { 2025-10-10T01:24:42.1515475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1515757Z "size": 724, 2025-10-10T01:24:42.1516032Z "digest": "sha256:3e62aac2efe8642154346954eca23d31a01d7d6558ac7f6c2f6ebdc2a2918ec0" 2025-10-10T01:24:42.1516355Z }, 2025-10-10T01:24:42.1516490Z { 2025-10-10T01:24:42.1516710Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1516990Z "size": 543, 2025-10-10T01:24:42.1517274Z "digest": "sha256:3c1f345eea91d1915d7438b5d8f77b9bcec1ced3dfff7d3ed56f622a50944ba4" 2025-10-10T01:24:42.1517600Z }, 2025-10-10T01:24:42.1517736Z { 2025-10-10T01:24:42.1517957Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1518252Z "size": 3412426191, 2025-10-10T01:24:42.1518541Z "digest": "sha256:a03ec53614528e4964121dfd061df88348df0558d1f8847d293703fb307e8565" 2025-10-10T01:24:42.1518986Z }, 2025-10-10T01:24:42.1519125Z { 2025-10-10T01:24:42.1519349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1519623Z "size": 32, 2025-10-10T01:24:42.1519904Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1520232Z }, 2025-10-10T01:24:42.1520366Z { 2025-10-10T01:24:42.1520580Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1520990Z "size": 380, 2025-10-10T01:24:42.1521276Z "digest": "sha256:84681debfb5c07ea57fc358c704ffa8b63dbcaf76db8e2dd7b12131db63760e1" 2025-10-10T01:24:42.1521614Z }, 2025-10-10T01:24:42.1521751Z { 2025-10-10T01:24:42.1521984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1522277Z "size": 236058, 2025-10-10T01:24:42.1522571Z "digest": "sha256:d6a77cb34f2d5411c23ad48da2ff399a5f331e2060be3c8cef0711829c69c696" 2025-10-10T01:24:42.1522898Z }, 2025-10-10T01:24:42.1523033Z { 2025-10-10T01:24:42.1523254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1523535Z "size": 231, 2025-10-10T01:24:42.1523802Z "digest": "sha256:b4e994693c0386d6b5a32dbc8f02222c10e8029b5714c7a9a77c04d0555a3cb5" 2025-10-10T01:24:42.1524118Z }, 2025-10-10T01:24:42.1524248Z { 2025-10-10T01:24:42.1524466Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1524756Z "size": 3548009, 2025-10-10T01:24:42.1525053Z "digest": "sha256:c6c1bfa3b4c534eaa64a0e9b78533c1c31e6368ceb0f5a8ae5362e44af9a8ee1" 2025-10-10T01:24:42.1525386Z }, 2025-10-10T01:24:42.1525517Z { 2025-10-10T01:24:42.1525736Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1526021Z "size": 1479, 2025-10-10T01:24:42.1526303Z "digest": "sha256:eb61ec0f4c410703c6a1d07f978c61c268b52bfb0e5f5139320f4dab72299fd3" 2025-10-10T01:24:42.1526624Z }, 2025-10-10T01:24:42.1526757Z { 2025-10-10T01:24:42.1526971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1527251Z "size": 482, 2025-10-10T01:24:42.1527527Z "digest": "sha256:31c940b54575a2a3c75778d62e786df5f6ea171157fe1a3301e9d44ecd63be4a" 2025-10-10T01:24:42.1527841Z }, 2025-10-10T01:24:42.1527968Z { 2025-10-10T01:24:42.1528191Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1528481Z "size": 199, 2025-10-10T01:24:42.1528764Z "digest": "sha256:21e8a5f2f65c173feee790b44ba3e36faef391dbe80291609660503d27c876b7" 2025-10-10T01:24:42.1529082Z }, 2025-10-10T01:24:42.1529218Z { 2025-10-10T01:24:42.1529530Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1529820Z "size": 608, 2025-10-10T01:24:42.1530094Z "digest": "sha256:496894807e5ced9b0cfe44157347c1d3369954a2c03be9c8fc7902aa87e83c2f" 2025-10-10T01:24:42.1530419Z }, 2025-10-10T01:24:42.1530553Z { 2025-10-10T01:24:42.1530774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1531049Z "size": 7871294039, 2025-10-10T01:24:42.1531342Z "digest": "sha256:1ae67e80b1bd70abd4d0cc28530080b079fe9851915401fc4efd12b0d30804e3" 2025-10-10T01:24:42.1531662Z }, 2025-10-10T01:24:42.1531794Z { 2025-10-10T01:24:42.1532007Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1532301Z "size": 829, 2025-10-10T01:24:42.1532575Z "digest": "sha256:cf225908c05b333296c57368e0e6e51ae79656d0396429e861119103f6dbe42e" 2025-10-10T01:24:42.1532883Z }, 2025-10-10T01:24:42.1533007Z { 2025-10-10T01:24:42.1533225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1533518Z "size": 33451742, 2025-10-10T01:24:42.1533815Z "digest": "sha256:381895ce55f63b7ce900d9f5a6488bf96bb1844eedb4daca0d2ec85c7b2a9d50" 2025-10-10T01:24:42.1534130Z }, 2025-10-10T01:24:42.1534262Z { 2025-10-10T01:24:42.1534480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1534761Z "size": 104, 2025-10-10T01:24:42.1535016Z "digest": "sha256:9263c0b02502319d82502d91c88966066495c32552fc2f0e3346a06a479029e9" 2025-10-10T01:24:42.1535319Z }, 2025-10-10T01:24:42.1535450Z { 2025-10-10T01:24:42.1535667Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1535942Z "size": 1494, 2025-10-10T01:24:42.1536229Z "digest": "sha256:df861c28c3a5b031b5566700eceda3adf67fed23ee9a24b470b2655bf58ebd6e" 2025-10-10T01:24:42.1536637Z }, 2025-10-10T01:24:42.1536767Z { 2025-10-10T01:24:42.1536979Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1537264Z "size": 453967971, 2025-10-10T01:24:42.1537554Z "digest": "sha256:72757cf9a787c2d65d34892517a1393a5006e11d113414fa1f648eb76a030030" 2025-10-10T01:24:42.1537870Z }, 2025-10-10T01:24:42.1537995Z { 2025-10-10T01:24:42.1538214Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1538490Z "size": 165, 2025-10-10T01:24:42.1538765Z "digest": "sha256:6ceacb336387b265adaf889cb35b12576e703638c547d104aab59905da34e10c" 2025-10-10T01:24:42.1539073Z }, 2025-10-10T01:24:42.1539214Z { 2025-10-10T01:24:42.1539443Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1539729Z "size": 346, 2025-10-10T01:24:42.1540027Z "digest": "sha256:f75cb5b656f8d33cc03600ad8f842daa41eee9ab78e963b982cc73e6f5a9d5a4" 2025-10-10T01:24:42.1540359Z }, 2025-10-10T01:24:42.1540494Z { 2025-10-10T01:24:42.1540714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1540995Z "size": 32, 2025-10-10T01:24:42.1541282Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1541606Z }, 2025-10-10T01:24:42.1541733Z { 2025-10-10T01:24:42.1541955Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1542238Z "size": 106, 2025-10-10T01:24:42.1542512Z "digest": "sha256:34e0c35a989365ff6b8bba7ef0a2454944f42d0552b30e564cd29058ed17e922" 2025-10-10T01:24:42.1542829Z }, 2025-10-10T01:24:42.1542961Z { 2025-10-10T01:24:42.1543182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1543476Z "size": 425, 2025-10-10T01:24:42.1543750Z "digest": "sha256:a3022990e5c36bc6dd359bbd4a75577a8afb5eab5f56d246c8365355f45c896d" 2025-10-10T01:24:42.1544074Z }, 2025-10-10T01:24:42.1544202Z { 2025-10-10T01:24:42.1544417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1544693Z "size": 19309421, 2025-10-10T01:24:42.1545058Z "digest": "sha256:93ef959930dc30c9d3829d76c89a3e12a1accde4f296210e73a923775466f6f3" 2025-10-10T01:24:42.1545378Z }, 2025-10-10T01:24:42.1545508Z { 2025-10-10T01:24:42.1545718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1545995Z "size": 108, 2025-10-10T01:24:42.1546264Z "digest": "sha256:fa921863e6e3c4729e4c156241dce0e0e91f06c58137c5a95e23366980f8dc48" 2025-10-10T01:24:42.1546575Z }, 2025-10-10T01:24:42.1546697Z { 2025-10-10T01:24:42.1546912Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1547189Z "size": 639, 2025-10-10T01:24:42.1547468Z "digest": "sha256:bab2c667cc1aba875b52e122d6ff86fa80cadb41c3060b4896423fdd41be3bf7" 2025-10-10T01:24:42.1547789Z }, 2025-10-10T01:24:42.1547918Z { 2025-10-10T01:24:42.1548156Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1548439Z "size": 724, 2025-10-10T01:24:42.1548709Z "digest": "sha256:3e62aac2efe8642154346954eca23d31a01d7d6558ac7f6c2f6ebdc2a2918ec0" 2025-10-10T01:24:42.1549032Z }, 2025-10-10T01:24:42.1549163Z { 2025-10-10T01:24:42.1549383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1549653Z "size": 149, 2025-10-10T01:24:42.1549928Z "digest": "sha256:3bc21d940f02abbc9ff39086f9d66c1e483b760b25c00efd69260f32a0f21bc3" 2025-10-10T01:24:42.1550248Z }, 2025-10-10T01:24:42.1550379Z { 2025-10-10T01:24:42.1550589Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1550869Z "size": 136, 2025-10-10T01:24:42.1551150Z "digest": "sha256:5a5fb6edd17b04afbe57dfd2bc9e4f92067ed6b9509b8a28a274c8802f7577a8" 2025-10-10T01:24:42.1551480Z }, 2025-10-10T01:24:42.1551609Z { 2025-10-10T01:24:42.1551831Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1552194Z "size": 140, 2025-10-10T01:24:42.1552487Z "digest": "sha256:62cb56db3356b895eb1bfc6a9352feafb0ba44e2e4bbbb933ee31466a89dc91f" 2025-10-10T01:24:42.1552814Z }, 2025-10-10T01:24:42.1552946Z { 2025-10-10T01:24:42.1553173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1553453Z "size": 32, 2025-10-10T01:24:42.1553727Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1554051Z }, 2025-10-10T01:24:42.1554185Z { 2025-10-10T01:24:42.1554403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1554676Z "size": 223, 2025-10-10T01:24:42.1554952Z "digest": "sha256:a1fff40729fd605a37158c121092ced87d96efdb3dad96d4b27f9e99d9e3a25b" 2025-10-10T01:24:42.1555271Z }, 2025-10-10T01:24:42.1555402Z { 2025-10-10T01:24:42.1555611Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1555900Z "size": 312468897, 2025-10-10T01:24:42.1556183Z "digest": "sha256:50e3f0227909c5eb4b1e149f9835f43ca029022668ca5d00f48f3ed1dc1d72ea" 2025-10-10T01:24:42.1556493Z }, 2025-10-10T01:24:42.1556618Z { 2025-10-10T01:24:42.1556840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1557121Z "size": 3275597028, 2025-10-10T01:24:42.1557413Z "digest": "sha256:ec62e2e53434b9077ac0ca94ade12194f7e0e4d1c9413fc114875ede7630178d" 2025-10-10T01:24:42.1557728Z }, 2025-10-10T01:24:42.1557860Z { 2025-10-10T01:24:42.1558078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1558369Z "size": 128, 2025-10-10T01:24:42.1558640Z "digest": "sha256:a47c72a0c062803832824190bdf9729fff33877e9e74ee333ba59eedbf97480d" 2025-10-10T01:24:42.1559025Z }, 2025-10-10T01:24:42.1559160Z { 2025-10-10T01:24:42.1559377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1559654Z "size": 880, 2025-10-10T01:24:42.1559931Z "digest": "sha256:04fa9fbfdbbdeb9977625cdcd35321e246859432e4cdd6980b7c4b23009aaebe" 2025-10-10T01:24:42.1560255Z }, 2025-10-10T01:24:42.1560385Z { 2025-10-10T01:24:42.1560679Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1560960Z "size": 724, 2025-10-10T01:24:42.1561232Z "digest": "sha256:3e62aac2efe8642154346954eca23d31a01d7d6558ac7f6c2f6ebdc2a2918ec0" 2025-10-10T01:24:42.1561551Z }, 2025-10-10T01:24:42.1561676Z { 2025-10-10T01:24:42.1561891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1562168Z "size": 141, 2025-10-10T01:24:42.1562439Z "digest": "sha256:df1cf69f3334c7338a0642cd84e318daf1a79280116974c76b9cc082096f5a8f" 2025-10-10T01:24:42.1562755Z }, 2025-10-10T01:24:42.1562881Z { 2025-10-10T01:24:42.1563111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1563394Z "size": 32, 2025-10-10T01:24:42.1563674Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1563997Z }, 2025-10-10T01:24:42.1564132Z { 2025-10-10T01:24:42.1564352Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1564637Z "size": 160, 2025-10-10T01:24:42.1564902Z "digest": "sha256:6ad7c264e84319c670a7ee23b32108cf557c91b12e8438059b6b7cc4cb3b1e92" 2025-10-10T01:24:42.1565218Z }, 2025-10-10T01:24:42.1565354Z { 2025-10-10T01:24:42.1565571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1565847Z "size": 1012, 2025-10-10T01:24:42.1566122Z "digest": "sha256:a08c722c1447f6cfb770780a5e9e0123d27e8e994f6c57b352fa71bb6c40d6f5" 2025-10-10T01:24:42.1566438Z }, 2025-10-10T01:24:42.1566570Z { 2025-10-10T01:24:42.1566778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1567066Z "size": 724, 2025-10-10T01:24:42.1567342Z "digest": "sha256:3e62aac2efe8642154346954eca23d31a01d7d6558ac7f6c2f6ebdc2a2918ec0" 2025-10-10T01:24:42.1567752Z }, 2025-10-10T01:24:42.1567875Z { 2025-10-10T01:24:42.1568090Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1568367Z "size": 135, 2025-10-10T01:24:42.1568653Z "digest": "sha256:38deb86def9c0f4fe784a0b2986dab1a87bbbafdf2464831502c7fddc595d2fc" 2025-10-10T01:24:42.1568975Z }, 2025-10-10T01:24:42.1569103Z { 2025-10-10T01:24:42.1569316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1569591Z "size": 32, 2025-10-10T01:24:42.1569861Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1570181Z }, 2025-10-10T01:24:42.1570311Z { 2025-10-10T01:24:42.1570525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1570797Z "size": 158, 2025-10-10T01:24:42.1571071Z "digest": "sha256:8a6cf39f45d150cd87db5867fe2dce8a814326a583db1f951f115e7a17fc6a27" 2025-10-10T01:24:42.1571397Z }, 2025-10-10T01:24:42.1571528Z { 2025-10-10T01:24:42.1571756Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1572036Z "size": 1369, 2025-10-10T01:24:42.1572312Z "digest": "sha256:a37d784168133fb9c2f7514a9fe459f85805b1717cc33187d3eb30a9f00eeccf" 2025-10-10T01:24:42.1572627Z }, 2025-10-10T01:24:42.1572756Z { 2025-10-10T01:24:42.1572976Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1573254Z "size": 32, 2025-10-10T01:24:42.1573528Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1573841Z }, 2025-10-10T01:24:42.1573972Z { 2025-10-10T01:24:42.1574192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1574469Z "size": 136, 2025-10-10T01:24:42.1574747Z "digest": "sha256:001023e5b31126d09bd1d68c21a7fae99118967088d1d825c0db18b0968cf4a6" 2025-10-10T01:24:42.1575060Z }, 2025-10-10T01:24:42.1575202Z { 2025-10-10T01:24:42.1575422Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1575696Z "size": 528, 2025-10-10T01:24:42.1575970Z "digest": "sha256:5b84bc540263a168bff65155a8a267e3cc7af2274fcfe84ba2d50776cb1f69a7" 2025-10-10T01:24:42.1576374Z }, 2025-10-10T01:24:42.1576516Z { 2025-10-10T01:24:42.1576730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1577010Z "size": 32, 2025-10-10T01:24:42.1577285Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1577604Z }, 2025-10-10T01:24:42.1577728Z { 2025-10-10T01:24:42.1577941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1578218Z "size": 104, 2025-10-10T01:24:42.1578504Z "digest": "sha256:7b03cfd9cf3c75c07836bf90040e0afd487cbd07cd3752e9a39f946e1726bf20" 2025-10-10T01:24:42.1578819Z }, 2025-10-10T01:24:42.1578950Z { 2025-10-10T01:24:42.1579170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1579453Z "size": 430, 2025-10-10T01:24:42.1579725Z "digest": "sha256:3ed8e92a019016b1a29fbc84a0d1fba5fc2495e81c5d84deec3c5a0be97fb1b0" 2025-10-10T01:24:42.1580046Z }, 2025-10-10T01:24:42.1580183Z { 2025-10-10T01:24:42.1580414Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1580689Z "size": 32, 2025-10-10T01:24:42.1580966Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1581284Z }, 2025-10-10T01:24:42.1581415Z { 2025-10-10T01:24:42.1581627Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1581902Z "size": 109, 2025-10-10T01:24:42.1582175Z "digest": "sha256:477967cba78d4f983881e91fdfb858a0f628e7533cf303e7cbe80f4c99deb43f" 2025-10-10T01:24:42.1582492Z }, 2025-10-10T01:24:42.1582616Z { 2025-10-10T01:24:42.1582833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1583190Z "size": 1897, 2025-10-10T01:24:42.1583470Z "digest": "sha256:6fffed0cb29593ef2bfbb6035811935c5b2a254e4467e360e058f40d5944d261" 2025-10-10T01:24:42.1583789Z }, 2025-10-10T01:24:42.1583915Z { 2025-10-10T01:24:42.1584148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1584438Z "size": 244514917, 2025-10-10T01:24:42.1584727Z "digest": "sha256:8aa311eb036b9b7ab88d5f00770ec0154cd7705a370477c53454e48ad49949c7" 2025-10-10T01:24:42.1585036Z }, 2025-10-10T01:24:42.1585168Z { 2025-10-10T01:24:42.1585385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1585663Z "size": 106, 2025-10-10T01:24:42.1585925Z "digest": "sha256:33d9650746499f91b89de8ab6ba0f6808dfa248eb37709a1686c3b57a07d0cc7" 2025-10-10T01:24:42.1586235Z }, 2025-10-10T01:24:42.1586362Z { 2025-10-10T01:24:42.1586574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1586846Z "size": 164, 2025-10-10T01:24:42.1587121Z "digest": "sha256:dbf89dacc6c6b267a7cd9200c81200a16386a20727ebfe38d3b05cfb223022e4" 2025-10-10T01:24:42.1587439Z }, 2025-10-10T01:24:42.1587568Z { 2025-10-10T01:24:42.1587785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1588076Z "size": 7943, 2025-10-10T01:24:42.1588350Z "digest": "sha256:208b50d13dad68e60745da16f1f48474ec502e57b6d3a49c863d432fbfc8eb5b" 2025-10-10T01:24:42.1588659Z }, 2025-10-10T01:24:42.1588781Z { 2025-10-10T01:24:42.1589001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1589282Z "size": 8070, 2025-10-10T01:24:42.1589562Z "digest": "sha256:863fa5fce8b803292f021ae5db81240ea781fdacdb093cd08202d4059b13535e" 2025-10-10T01:24:42.1589883Z }, 2025-10-10T01:24:42.1590010Z { 2025-10-10T01:24:42.1590231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1590511Z "size": 303, 2025-10-10T01:24:42.1590787Z "digest": "sha256:2c57470f9b051ed7b272e9cb78b567b3a9672312e2620dbe5432a70d95bb206b" 2025-10-10T01:24:42.1591097Z }, 2025-10-10T01:24:42.1591229Z { 2025-10-10T01:24:42.1591449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1591813Z "size": 13360730, 2025-10-10T01:24:42.1592105Z "digest": "sha256:4fbac50fbb3bb3f8695474fefc6fe9230fc5c628ab961fc42c5b8ad275f1f2d5" 2025-10-10T01:24:42.1592441Z }, 2025-10-10T01:24:42.1592575Z { 2025-10-10T01:24:42.1592793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1593065Z "size": 108, 2025-10-10T01:24:42.1593336Z "digest": "sha256:6baf3cf73050abcdf92034c01cb77c2d6467688d409f22773e8c21e93ed76a42" 2025-10-10T01:24:42.1593651Z }, 2025-10-10T01:24:42.1593780Z { 2025-10-10T01:24:42.1593989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1594265Z "size": 54145699, 2025-10-10T01:24:42.1594548Z "digest": "sha256:ace2a84685d29b76147a66dfa220c800615a65949511de6b464aff673eeaf062" 2025-10-10T01:24:42.1594869Z }, 2025-10-10T01:24:42.1594993Z { 2025-10-10T01:24:42.1595208Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T01:24:42.1595483Z "size": 32, 2025-10-10T01:24:42.1595765Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T01:24:42.1596339Z } 2025-10-10T01:24:42.1596520Z ] 2025-10-10T01:24:42.1596656Z } 2025-10-10T01:24:42.1627782Z ##[group]Run set -eux 2025-10-10T01:24:42.1627991Z set -eux 2025-10-10T01:24:42.1628292Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-10-10T01:24:42.1629103Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-10-10T01:24:42.1637609Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:42.1638036Z env: 2025-10-10T01:24:42.1638201Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:42.1638392Z ##[endgroup] 2025-10-10T01:24:42.1670879Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-10-10T01:24:42.1672223Z + jq --raw-output .SecretString 2025-10-10T01:24:42.1673357Z + jq -r .docker_hub_readonly_token 2025-10-10T01:24:42.1674592Z + docker login --username pytorchbot --password-stdin 2025-10-10T01:24:42.6832731Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T01:24:42.6833311Z Configure a credential helper to remove this warning. See 2025-10-10T01:24:42.6833845Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T01:24:42.6834197Z 2025-10-10T01:24:42.6834509Z Login Succeeded 2025-10-10T01:24:42.6927039Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-10-10T01:24:42.6927346Z tag=${ECR_DOCKER_IMAGE##*:} 2025-10-10T01:24:42.6927653Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-10-10T01:24:42.6935651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:42.6935939Z env: 2025-10-10T01:24:42.6936103Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:42.6936727Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:42.6937362Z ##[endgroup] 2025-10-10T01:24:42.6968195Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:42.7019466Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-10-10T01:24:42.7019816Z with: 2025-10-10T01:24:42.7020396Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:42.7021080Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:42.7021362Z env: 2025-10-10T01:24:42.7021517Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:42.7021703Z ##[endgroup] 2025-10-10T01:24:42.7053571Z ##[group]Run set -x 2025-10-10T01:24:42.7053782Z set -x 2025-10-10T01:24:42.7053948Z set +e 2025-10-10T01:24:42.7054115Z  2025-10-10T01:24:42.7054264Z login() { 2025-10-10T01:24:42.7054613Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-10-10T01:24:42.7054983Z } 2025-10-10T01:24:42.7055130Z  2025-10-10T01:24:42.7055319Z retry () { 2025-10-10T01:24:42.7055518Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-10-10T01:24:42.7055742Z } 2025-10-10T01:24:42.7055890Z  2025-10-10T01:24:42.7056058Z retry login "${DOCKER_REGISTRY}" 2025-10-10T01:24:42.7056278Z  2025-10-10T01:24:42.7056617Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-10-10T01:24:42.7057087Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-10-10T01:24:42.7057361Z  2025-10-10T01:24:42.7057513Z set -e 2025-10-10T01:24:42.7057763Z # ignore output since only exit code is used for conditional 2025-10-10T01:24:42.7058111Z # only pull docker image if it's not available locally 2025-10-10T01:24:42.7058487Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-10-10T01:24:42.7058841Z  retry docker pull "${DOCKER_IMAGE}" 2025-10-10T01:24:42.7059068Z fi 2025-10-10T01:24:42.7065995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:24:42.7066283Z env: 2025-10-10T01:24:42.7066443Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:24:42.7067054Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:42.7067916Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:42.7068191Z ##[endgroup] 2025-10-10T01:24:42.7106658Z + set +e 2025-10-10T01:24:42.7106972Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:42.7107360Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:42.7110196Z + aws ecr get-login-password --region us-east-1 2025-10-10T01:24:42.7111403Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T01:24:43.1731548Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T01:24:43.1732105Z Configure a credential helper to remove this warning. See 2025-10-10T01:24:43.1732615Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T01:24:43.1732956Z 2025-10-10T01:24:43.1733693Z Login Succeeded 2025-10-10T01:24:43.1756802Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:43.1757706Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-10-10T01:24:43.3878844Z + IMAGE_SIZE=15400.734984397888 2025-10-10T01:24:43.3879351Z + echo 'Compressed size of image in MB: 15400.734984397888' 2025-10-10T01:24:43.3879696Z + set -e 2025-10-10T01:24:43.3880821Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:43.3881752Z Compressed size of image in MB: 15400.734984397888 2025-10-10T01:24:43.4012900Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:43.4014264Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:24:43.6733837Z pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac: Pulling from pytorch/ci-image 2025-10-10T01:24:43.6734977Z 828c1365039a: Pulling fs layer 2025-10-10T01:24:43.6735327Z 140bb29742fd: Pulling fs layer 2025-10-10T01:24:43.6735674Z 99ee2db1caf9: Pulling fs layer 2025-10-10T01:24:43.6736043Z f5a2330f87a8: Pulling fs layer 2025-10-10T01:24:43.6736362Z 4643d3cb7e67: Pulling fs layer 2025-10-10T01:24:43.6736709Z 5c871f60f6d4: Pulling fs layer 2025-10-10T01:24:43.6737035Z 60a2970be007: Pulling fs layer 2025-10-10T01:24:43.6737357Z 43d427ee0b00: Pulling fs layer 2025-10-10T01:24:43.6737666Z 728b6e1eb012: Pulling fs layer 2025-10-10T01:24:43.6738001Z 1ffc73eae77b: Pulling fs layer 2025-10-10T01:24:43.6738315Z 1c5379881a3b: Pulling fs layer 2025-10-10T01:24:43.6738644Z 3e62aac2efe8: Pulling fs layer 2025-10-10T01:24:43.6738973Z 3c1f345eea91: Pulling fs layer 2025-10-10T01:24:43.6739292Z a03ec5361452: Pulling fs layer 2025-10-10T01:24:43.6739619Z 4f4fb700ef54: Pulling fs layer 2025-10-10T01:24:43.6739935Z 84681debfb5c: Pulling fs layer 2025-10-10T01:24:43.6740244Z d6a77cb34f2d: Pulling fs layer 2025-10-10T01:24:43.6740558Z b4e994693c03: Pulling fs layer 2025-10-10T01:24:43.6740881Z c6c1bfa3b4c5: Pulling fs layer 2025-10-10T01:24:43.6741192Z eb61ec0f4c41: Pulling fs layer 2025-10-10T01:24:43.6741497Z 31c940b54575: Pulling fs layer 2025-10-10T01:24:43.6741819Z 21e8a5f2f65c: Pulling fs layer 2025-10-10T01:24:43.6742127Z 496894807e5c: Pulling fs layer 2025-10-10T01:24:43.6742429Z 728b6e1eb012: Waiting 2025-10-10T01:24:43.6742698Z 1ffc73eae77b: Waiting 2025-10-10T01:24:43.6742990Z 1ae67e80b1bd: Pulling fs layer 2025-10-10T01:24:43.6743316Z cf225908c05b: Pulling fs layer 2025-10-10T01:24:43.6743638Z 381895ce55f6: Pulling fs layer 2025-10-10T01:24:43.6743949Z 9263c0b02502: Pulling fs layer 2025-10-10T01:24:43.6744255Z 1c5379881a3b: Waiting 2025-10-10T01:24:43.6744801Z df861c28c3a5: Pulling fs layer 2025-10-10T01:24:43.6745129Z 72757cf9a787: Pulling fs layer 2025-10-10T01:24:43.6745436Z 6ceacb336387: Pulling fs layer 2025-10-10T01:24:43.6745753Z f75cb5b656f8: Pulling fs layer 2025-10-10T01:24:43.6746092Z 34e0c35a9893: Pulling fs layer 2025-10-10T01:24:43.6746395Z a3022990e5c3: Pulling fs layer 2025-10-10T01:24:43.6746702Z 93ef959930dc: Pulling fs layer 2025-10-10T01:24:43.6746993Z 4f4fb700ef54: Waiting 2025-10-10T01:24:43.6747435Z fa921863e6e3: Pulling fs layer 2025-10-10T01:24:43.6747767Z bab2c667cc1a: Pulling fs layer 2025-10-10T01:24:43.6748102Z 3bc21d940f02: Pulling fs layer 2025-10-10T01:24:43.6748287Z c6c1bfa3b4c5: Waiting 2025-10-10T01:24:43.6748472Z 5a5fb6edd17b: Pulling fs layer 2025-10-10T01:24:43.6748661Z 62cb56db3356: Pulling fs layer 2025-10-10T01:24:43.6748849Z a1fff40729fd: Pulling fs layer 2025-10-10T01:24:43.6749026Z eb61ec0f4c41: Waiting 2025-10-10T01:24:43.6749243Z 31c940b54575: Waiting 2025-10-10T01:24:43.6749421Z 21e8a5f2f65c: Waiting 2025-10-10T01:24:43.6749588Z 50e3f0227909: Pulling fs layer 2025-10-10T01:24:43.6749774Z ec62e2e53434: Pulling fs layer 2025-10-10T01:24:43.6749955Z 496894807e5c: Waiting 2025-10-10T01:24:43.6750117Z cf225908c05b: Waiting 2025-10-10T01:24:43.6750279Z 9263c0b02502: Waiting 2025-10-10T01:24:43.6750440Z a47c72a0c062: Pulling fs layer 2025-10-10T01:24:43.6750626Z 84681debfb5c: Waiting 2025-10-10T01:24:43.6750798Z 04fa9fbfdbbd: Pulling fs layer 2025-10-10T01:24:43.6750992Z df1cf69f3334: Pulling fs layer 2025-10-10T01:24:43.6751170Z 1ae67e80b1bd: Waiting 2025-10-10T01:24:43.6751330Z b4e994693c03: Waiting 2025-10-10T01:24:43.6751682Z 6ad7c264e843: Pulling fs layer 2025-10-10T01:24:43.6751883Z 4643d3cb7e67: Waiting 2025-10-10T01:24:43.6752037Z a3022990e5c3: Waiting 2025-10-10T01:24:43.6752191Z 34e0c35a9893: Waiting 2025-10-10T01:24:43.6752367Z a08c722c1447: Pulling fs layer 2025-10-10T01:24:43.6752556Z 38deb86def9c: Pulling fs layer 2025-10-10T01:24:43.6752731Z 6ceacb336387: Waiting 2025-10-10T01:24:43.6752898Z 8a6cf39f45d1: Pulling fs layer 2025-10-10T01:24:43.6753082Z a37d78416813: Pulling fs layer 2025-10-10T01:24:43.6753251Z 60a2970be007: Waiting 2025-10-10T01:24:43.6753407Z fa921863e6e3: Waiting 2025-10-10T01:24:43.6753562Z 5c871f60f6d4: Waiting 2025-10-10T01:24:43.6753721Z bab2c667cc1a: Waiting 2025-10-10T01:24:43.6753872Z 3bc21d940f02: Waiting 2025-10-10T01:24:43.6754041Z 43d427ee0b00: Waiting 2025-10-10T01:24:43.6754206Z 001023e5b311: Pulling fs layer 2025-10-10T01:24:43.6754382Z 93ef959930dc: Waiting 2025-10-10T01:24:43.6754534Z a1fff40729fd: Waiting 2025-10-10T01:24:43.6754693Z f75cb5b656f8: Waiting 2025-10-10T01:24:43.6754858Z 5b84bc540263: Pulling fs layer 2025-10-10T01:24:43.6755038Z 62cb56db3356: Waiting 2025-10-10T01:24:43.6755187Z 50e3f0227909: Waiting 2025-10-10T01:24:43.6755352Z 7b03cfd9cf3c: Pulling fs layer 2025-10-10T01:24:43.6755552Z 3ed8e92a0190: Pulling fs layer 2025-10-10T01:24:43.6755735Z df1cf69f3334: Waiting 2025-10-10T01:24:43.6755894Z 477967cba78d: Pulling fs layer 2025-10-10T01:24:43.6756077Z ec62e2e53434: Waiting 2025-10-10T01:24:43.6756232Z 6ad7c264e843: Waiting 2025-10-10T01:24:43.6756390Z 04fa9fbfdbbd: Waiting 2025-10-10T01:24:43.6756563Z 6fffed0cb295: Pulling fs layer 2025-10-10T01:24:43.6756748Z f5a2330f87a8: Waiting 2025-10-10T01:24:43.6756914Z 8aa311eb036b: Pulling fs layer 2025-10-10T01:24:43.6757100Z 33d965074649: Pulling fs layer 2025-10-10T01:24:43.6757281Z dbf89dacc6c6: Pulling fs layer 2025-10-10T01:24:43.6757462Z a47c72a0c062: Waiting 2025-10-10T01:24:43.6757629Z 208b50d13dad: Pulling fs layer 2025-10-10T01:24:43.6757808Z 3c1f345eea91: Waiting 2025-10-10T01:24:43.6757977Z 001023e5b311: Waiting 2025-10-10T01:24:43.6758145Z 863fa5fce8b8: Pulling fs layer 2025-10-10T01:24:43.6758343Z 2c57470f9b05: Pulling fs layer 2025-10-10T01:24:43.6758533Z 4fbac50fbb3b: Pulling fs layer 2025-10-10T01:24:43.6758709Z 5b84bc540263: Waiting 2025-10-10T01:24:43.6759369Z 6baf3cf73050: Pulling fs layer 2025-10-10T01:24:43.6759604Z 3ed8e92a0190: Waiting 2025-10-10T01:24:43.6759913Z ace2a84685d2: Pulling fs layer 2025-10-10T01:24:43.6760099Z 6fffed0cb295: Waiting 2025-10-10T01:24:43.6760267Z 477967cba78d: Waiting 2025-10-10T01:24:43.6760439Z a08c722c1447: Waiting 2025-10-10T01:24:43.6760606Z 4fbac50fbb3b: Waiting 2025-10-10T01:24:43.6760760Z 8aa311eb036b: Waiting 2025-10-10T01:24:43.6760929Z 33d965074649: Waiting 2025-10-10T01:24:43.6761086Z 6baf3cf73050: Waiting 2025-10-10T01:24:43.6761245Z dbf89dacc6c6: Waiting 2025-10-10T01:24:43.6761397Z 208b50d13dad: Waiting 2025-10-10T01:24:43.6761615Z 863fa5fce8b8: Waiting 2025-10-10T01:24:43.6761786Z ace2a84685d2: Waiting 2025-10-10T01:24:43.6761943Z 2c57470f9b05: Waiting 2025-10-10T01:24:43.6762100Z 8a6cf39f45d1: Waiting 2025-10-10T01:24:43.6762255Z a37d78416813: Waiting 2025-10-10T01:24:43.6762411Z 7b03cfd9cf3c: Waiting 2025-10-10T01:24:43.6762568Z 3e62aac2efe8: Waiting 2025-10-10T01:24:43.6762720Z d6a77cb34f2d: Waiting 2025-10-10T01:24:43.6762877Z 38deb86def9c: Waiting 2025-10-10T01:24:43.6763044Z df861c28c3a5: Waiting 2025-10-10T01:24:43.6763204Z 5a5fb6edd17b: Waiting 2025-10-10T01:24:43.6763424Z 381895ce55f6: Waiting 2025-10-10T01:24:43.6763580Z 72757cf9a787: Waiting 2025-10-10T01:24:43.6763734Z a03ec5361452: Waiting 2025-10-10T01:24:43.7685479Z 140bb29742fd: Verifying Checksum 2025-10-10T01:24:43.7685960Z 140bb29742fd: Download complete 2025-10-10T01:24:43.8548616Z f5a2330f87a8: Verifying Checksum 2025-10-10T01:24:43.8549058Z f5a2330f87a8: Download complete 2025-10-10T01:24:43.9240986Z 4643d3cb7e67: Verifying Checksum 2025-10-10T01:24:43.9241387Z 4643d3cb7e67: Download complete 2025-10-10T01:24:44.0088098Z 5c871f60f6d4: Download complete 2025-10-10T01:24:44.0179857Z 828c1365039a: Verifying Checksum 2025-10-10T01:24:44.0180331Z 828c1365039a: Download complete 2025-10-10T01:24:44.0983848Z 43d427ee0b00: Verifying Checksum 2025-10-10T01:24:44.0986472Z 43d427ee0b00: Download complete 2025-10-10T01:24:44.1081242Z 60a2970be007: Download complete 2025-10-10T01:24:44.2256375Z 1ffc73eae77b: Verifying Checksum 2025-10-10T01:24:44.2256691Z 1ffc73eae77b: Download complete 2025-10-10T01:24:44.3009452Z 1c5379881a3b: Download complete 2025-10-10T01:24:44.3892491Z 3e62aac2efe8: Verifying Checksum 2025-10-10T01:24:44.3892818Z 3e62aac2efe8: Download complete 2025-10-10T01:24:44.4815726Z 3c1f345eea91: Download complete 2025-10-10T01:24:44.9318817Z 828c1365039a: Pull complete 2025-10-10T01:24:44.9549979Z 140bb29742fd: Pull complete 2025-10-10T01:24:45.2630400Z 728b6e1eb012: Verifying Checksum 2025-10-10T01:24:45.2630848Z 728b6e1eb012: Download complete 2025-10-10T01:24:45.2720420Z 4f4fb700ef54: Verifying Checksum 2025-10-10T01:24:45.2720822Z 4f4fb700ef54: Download complete 2025-10-10T01:24:45.3569743Z 84681debfb5c: Verifying Checksum 2025-10-10T01:24:45.3570158Z 84681debfb5c: Download complete 2025-10-10T01:24:45.4476111Z d6a77cb34f2d: Download complete 2025-10-10T01:24:45.5308762Z b4e994693c03: Verifying Checksum 2025-10-10T01:24:45.5309166Z b4e994693c03: Download complete 2025-10-10T01:24:45.6172615Z c6c1bfa3b4c5: Verifying Checksum 2025-10-10T01:24:45.6173082Z c6c1bfa3b4c5: Download complete 2025-10-10T01:24:45.7234764Z eb61ec0f4c41: Verifying Checksum 2025-10-10T01:24:45.7235123Z eb61ec0f4c41: Download complete 2025-10-10T01:24:45.8101634Z 31c940b54575: Verifying Checksum 2025-10-10T01:24:45.8102041Z 31c940b54575: Download complete 2025-10-10T01:24:45.8858469Z 21e8a5f2f65c: Verifying Checksum 2025-10-10T01:24:45.8858801Z 21e8a5f2f65c: Download complete 2025-10-10T01:24:45.9691059Z 496894807e5c: Download complete 2025-10-10T01:24:46.8613506Z 99ee2db1caf9: Verifying Checksum 2025-10-10T01:24:46.8613877Z 99ee2db1caf9: Download complete 2025-10-10T01:24:46.9402448Z cf225908c05b: Verifying Checksum 2025-10-10T01:24:46.9402882Z cf225908c05b: Download complete 2025-10-10T01:24:47.3350446Z 381895ce55f6: Verifying Checksum 2025-10-10T01:24:47.3350888Z 381895ce55f6: Download complete 2025-10-10T01:24:47.4437466Z 9263c0b02502: Verifying Checksum 2025-10-10T01:24:47.4437824Z 9263c0b02502: Download complete 2025-10-10T01:24:47.5225394Z df861c28c3a5: Download complete 2025-10-10T01:24:52.1101634Z 72757cf9a787: Verifying Checksum 2025-10-10T01:24:52.1101960Z 72757cf9a787: Download complete 2025-10-10T01:24:52.2053067Z 6ceacb336387: Download complete 2025-10-10T01:24:52.2826108Z f75cb5b656f8: Verifying Checksum 2025-10-10T01:24:52.2826428Z f75cb5b656f8: Download complete 2025-10-10T01:24:52.3775077Z 34e0c35a9893: Verifying Checksum 2025-10-10T01:24:52.3775383Z 34e0c35a9893: Download complete 2025-10-10T01:24:52.4739865Z a3022990e5c3: Verifying Checksum 2025-10-10T01:24:52.4740307Z a3022990e5c3: Download complete 2025-10-10T01:24:52.7153550Z 93ef959930dc: Verifying Checksum 2025-10-10T01:24:52.7154003Z 93ef959930dc: Download complete 2025-10-10T01:24:52.7980341Z fa921863e6e3: Verifying Checksum 2025-10-10T01:24:52.7980682Z fa921863e6e3: Download complete 2025-10-10T01:24:52.8931098Z bab2c667cc1a: Verifying Checksum 2025-10-10T01:24:52.8931444Z bab2c667cc1a: Download complete 2025-10-10T01:24:52.9707903Z 3bc21d940f02: Verifying Checksum 2025-10-10T01:24:52.9708462Z 3bc21d940f02: Download complete 2025-10-10T01:24:53.0654536Z 5a5fb6edd17b: Download complete 2025-10-10T01:24:53.1372861Z 62cb56db3356: Download complete 2025-10-10T01:24:53.2189578Z a1fff40729fd: Verifying Checksum 2025-10-10T01:24:53.2190015Z a1fff40729fd: Download complete 2025-10-10T01:24:54.1118522Z 99ee2db1caf9: Pull complete 2025-10-10T01:24:54.1345523Z f5a2330f87a8: Pull complete 2025-10-10T01:24:54.1550038Z 4643d3cb7e67: Pull complete 2025-10-10T01:24:54.1758367Z 5c871f60f6d4: Pull complete 2025-10-10T01:24:54.2014553Z 60a2970be007: Pull complete 2025-10-10T01:24:54.2275059Z 43d427ee0b00: Pull complete 2025-10-10T01:24:56.2783435Z 728b6e1eb012: Pull complete 2025-10-10T01:24:56.4675905Z 1ffc73eae77b: Pull complete 2025-10-10T01:24:56.5682643Z 1c5379881a3b: Pull complete 2025-10-10T01:24:56.6524012Z 3e62aac2efe8: Pull complete 2025-10-10T01:24:56.7335435Z 3c1f345eea91: Pull complete 2025-10-10T01:25:00.3189734Z 50e3f0227909: Verifying Checksum 2025-10-10T01:25:00.3190035Z 50e3f0227909: Download complete 2025-10-10T01:25:18.8255016Z a03ec5361452: Verifying Checksum 2025-10-10T01:25:18.8255562Z a03ec5361452: Download complete 2025-10-10T01:25:18.8998686Z a47c72a0c062: Download complete 2025-10-10T01:25:18.9941361Z 04fa9fbfdbbd: Verifying Checksum 2025-10-10T01:25:19.0875702Z df1cf69f3334: Verifying Checksum 2025-10-10T01:25:19.0876195Z df1cf69f3334: Download complete 2025-10-10T01:25:19.1817305Z 6ad7c264e843: Verifying Checksum 2025-10-10T01:25:19.1817761Z 6ad7c264e843: Download complete 2025-10-10T01:25:19.2487535Z a08c722c1447: Download complete 2025-10-10T01:25:19.3252008Z 38deb86def9c: Verifying Checksum 2025-10-10T01:25:19.3252394Z 38deb86def9c: Download complete 2025-10-10T01:25:19.4020661Z 8a6cf39f45d1: Verifying Checksum 2025-10-10T01:25:19.4021130Z 8a6cf39f45d1: Download complete 2025-10-10T01:25:19.4747667Z a37d78416813: Verifying Checksum 2025-10-10T01:25:19.4748005Z a37d78416813: Download complete 2025-10-10T01:25:19.5432236Z 001023e5b311: Download complete 2025-10-10T01:25:19.6208107Z 5b84bc540263: Download complete 2025-10-10T01:25:19.7286606Z 7b03cfd9cf3c: Verifying Checksum 2025-10-10T01:25:19.7287121Z 7b03cfd9cf3c: Download complete 2025-10-10T01:25:19.8030252Z 3ed8e92a0190: Verifying Checksum 2025-10-10T01:25:19.8030703Z 3ed8e92a0190: Download complete 2025-10-10T01:25:19.8813130Z 477967cba78d: Verifying Checksum 2025-10-10T01:25:19.8813576Z 477967cba78d: Download complete 2025-10-10T01:25:19.9538720Z 6fffed0cb295: Verifying Checksum 2025-10-10T01:25:19.9539220Z 6fffed0cb295: Download complete 2025-10-10T01:25:22.4391613Z 8aa311eb036b: Verifying Checksum 2025-10-10T01:25:22.4391986Z 8aa311eb036b: Download complete 2025-10-10T01:25:22.5300592Z 33d965074649: Verifying Checksum 2025-10-10T01:25:22.5300941Z 33d965074649: Download complete 2025-10-10T01:25:22.5877577Z dbf89dacc6c6: Download complete 2025-10-10T01:25:22.6649949Z 208b50d13dad: Download complete 2025-10-10T01:25:22.7340831Z 863fa5fce8b8: Verifying Checksum 2025-10-10T01:25:22.7341239Z 863fa5fce8b8: Download complete 2025-10-10T01:25:22.8330763Z 2c57470f9b05: Verifying Checksum 2025-10-10T01:25:22.8331482Z 2c57470f9b05: Download complete 2025-10-10T01:25:23.0123897Z 4fbac50fbb3b: Download complete 2025-10-10T01:25:23.0843724Z 6baf3cf73050: Verifying Checksum 2025-10-10T01:25:23.0844098Z 6baf3cf73050: Download complete 2025-10-10T01:25:23.6693046Z ace2a84685d2: Verifying Checksum 2025-10-10T01:25:23.6693492Z ace2a84685d2: Download complete 2025-10-10T01:25:33.8009894Z ec62e2e53434: Verifying Checksum 2025-10-10T01:25:33.8010383Z ec62e2e53434: Download complete 2025-10-10T01:26:09.0943721Z 1ae67e80b1bd: Verifying Checksum 2025-10-10T01:26:09.0944091Z 1ae67e80b1bd: Download complete 2025-10-10T01:26:43.5147896Z a03ec5361452: Pull complete 2025-10-10T01:26:43.6053168Z 4f4fb700ef54: Pull complete 2025-10-10T01:26:43.7206471Z 84681debfb5c: Pull complete 2025-10-10T01:26:43.8711303Z d6a77cb34f2d: Pull complete 2025-10-10T01:26:44.0080774Z b4e994693c03: Pull complete 2025-10-10T01:26:44.1940079Z c6c1bfa3b4c5: Pull complete 2025-10-10T01:26:44.2457243Z eb61ec0f4c41: Pull complete 2025-10-10T01:26:44.3281856Z 31c940b54575: Pull complete 2025-10-10T01:26:44.4899655Z 21e8a5f2f65c: Pull complete 2025-10-10T01:26:44.6104622Z 496894807e5c: Pull complete 2025-10-10T01:28:18.7000887Z 1ae67e80b1bd: Pull complete 2025-10-10T01:28:18.9141908Z cf225908c05b: Pull complete 2025-10-10T01:28:19.5006055Z 381895ce55f6: Pull complete 2025-10-10T01:28:19.7163124Z 9263c0b02502: Pull complete 2025-10-10T01:28:19.7478593Z df861c28c3a5: Pull complete 2025-10-10T01:28:26.4214665Z 72757cf9a787: Pull complete 2025-10-10T01:28:26.6439801Z 6ceacb336387: Pull complete 2025-10-10T01:28:26.8609989Z f75cb5b656f8: Pull complete 2025-10-10T01:28:27.3095372Z 34e0c35a9893: Pull complete 2025-10-10T01:28:27.5273880Z a3022990e5c3: Pull complete 2025-10-10T01:28:27.8975197Z 93ef959930dc: Pull complete 2025-10-10T01:28:28.1185899Z fa921863e6e3: Pull complete 2025-10-10T01:28:28.3453591Z bab2c667cc1a: Pull complete 2025-10-10T01:28:28.7821726Z 3bc21d940f02: Pull complete 2025-10-10T01:28:29.0063930Z 5a5fb6edd17b: Pull complete 2025-10-10T01:28:29.2180466Z 62cb56db3356: Pull complete 2025-10-10T01:28:29.6604741Z a1fff40729fd: Pull complete 2025-10-10T01:28:30.7305584Z 50e3f0227909: Pull complete 2025-10-10T01:29:15.3482524Z ec62e2e53434: Pull complete 2025-10-10T01:29:15.5235074Z a47c72a0c062: Pull complete 2025-10-10T01:29:15.6576099Z 04fa9fbfdbbd: Pull complete 2025-10-10T01:29:15.9174638Z df1cf69f3334: Pull complete 2025-10-10T01:29:16.1437134Z 6ad7c264e843: Pull complete 2025-10-10T01:29:16.2823252Z a08c722c1447: Pull complete 2025-10-10T01:29:16.5397107Z 38deb86def9c: Pull complete 2025-10-10T01:29:16.8797529Z 8a6cf39f45d1: Pull complete 2025-10-10T01:29:17.0850840Z a37d78416813: Pull complete 2025-10-10T01:29:17.4171473Z 001023e5b311: Pull complete 2025-10-10T01:29:17.6140251Z 5b84bc540263: Pull complete 2025-10-10T01:29:18.0137291Z 7b03cfd9cf3c: Pull complete 2025-10-10T01:29:18.2248381Z 3ed8e92a0190: Pull complete 2025-10-10T01:29:18.5889738Z 477967cba78d: Pull complete 2025-10-10T01:29:18.7983025Z 6fffed0cb295: Pull complete 2025-10-10T01:29:24.4989184Z 8aa311eb036b: Pull complete 2025-10-10T01:29:24.7082883Z 33d965074649: Pull complete 2025-10-10T01:29:24.9218612Z dbf89dacc6c6: Pull complete 2025-10-10T01:29:25.1221403Z 208b50d13dad: Pull complete 2025-10-10T01:29:25.3331992Z 863fa5fce8b8: Pull complete 2025-10-10T01:29:25.5493317Z 2c57470f9b05: Pull complete 2025-10-10T01:29:26.9577194Z 4fbac50fbb3b: Pull complete 2025-10-10T01:29:27.1694853Z 6baf3cf73050: Pull complete 2025-10-10T01:29:28.4236485Z ace2a84685d2: Pull complete 2025-10-10T01:29:28.7045706Z Digest: sha256:dbf8617838c93f0ebd67b8b8bb938f5947e45ccc171ead9c6da61e625c32803f 2025-10-10T01:29:28.7476325Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:29:28.7724063Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:29:28.7774600Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T01:29:28.7775373Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T01:29:28.7786975Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:29:28.7787261Z env: 2025-10-10T01:29:28.7787425Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:29:28.7787620Z ##[endgroup] 2025-10-10T01:29:28.7963070Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-10-10T01:29:28.7963410Z with: 2025-10-10T01:29:28.7963580Z driver-version: 580.82.07 2025-10-10T01:29:28.7963774Z env: 2025-10-10T01:29:28.7963932Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:29:28.7964121Z ##[endgroup] 2025-10-10T01:29:28.8128503Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2025-10-10T01:29:28.8128833Z with: 2025-10-10T01:29:28.8128984Z timeout_minutes: 10 2025-10-10T01:29:28.8129206Z max_attempts: 3 2025-10-10T01:29:28.8148781Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y \ nvidia-container-toolkit-1.17.8 \ libnvidia-container-tools-1.17.8 \ libnvidia-container1-1.17.8 \ nvidia-container-toolkit-base-1.17.8 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-container-toolkit-1.17.8 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" # Turn off persistent mode so that the installation script can unload the kernel module sudo killall nvidia-persistenced || true else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi # check if the container-toolkit is correctly installed and CUDA is available inside a container docker run --rm -t --gpus=all public.ecr.aws/docker/library/python:3.13 nvidia-smi 2025-10-10T01:29:28.8168360Z retry_wait_seconds: 10 2025-10-10T01:29:28.8168573Z polling_interval_seconds: 1 2025-10-10T01:29:28.8168782Z warning_on_retry: true 2025-10-10T01:29:28.8168960Z continue_on_error: false 2025-10-10T01:29:28.8169141Z env: 2025-10-10T01:29:28.8169304Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:29:28.8169492Z DRIVER_VERSION: 580.82.07 2025-10-10T01:29:28.8169671Z ##[endgroup] 2025-10-10T01:29:28.9036933Z == Installing nvidia driver NVIDIA-Linux-x86_64-580.82.07.run == 2025-10-10T01:29:28.9038747Z + pre_install_nvidia_driver_amzn2 2025-10-10T01:29:28.9041175Z + sudo yum remove -y nvidia-driver-latest-dkms 2025-10-10T01:29:29.5458120Z No match for argument: nvidia-driver-latest-dkms 2025-10-10T01:29:29.5458773Z No packages marked for removal. 2025-10-10T01:29:29.5513221Z Dependencies resolved. 2025-10-10T01:29:29.5521980Z Nothing to do. 2025-10-10T01:29:29.5522701Z Complete! 2025-10-10T01:29:29.6260160Z + install_nvidia_driver_common 2025-10-10T01:29:29.6262187Z + echo 'Before installing NVIDIA driver' 2025-10-10T01:29:29.6263704Z Before installing NVIDIA driver 2025-10-10T01:29:29.6265734Z + lspci 2025-10-10T01:29:29.7389440Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-10-10T01:29:29.7389946Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-10-10T01:29:29.7390476Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-10-10T01:29:29.7391289Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-10-10T01:29:29.7391768Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-10-10T01:29:29.7392191Z 01:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7392530Z 02:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7392867Z 03:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7393186Z 03:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7393526Z 03:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7393797Z 03:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7394037Z 03:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7394279Z 03:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7394522Z 03:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7394759Z 03:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7395008Z 03:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7395252Z 03:01.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7395496Z 03:01.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7395729Z 03:01.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7396004Z 03:01.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7396477Z 03:01.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7396737Z 03:01.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7396985Z 03:01.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7397221Z 03:02.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7397458Z 03:02.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7397708Z 03:02.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7397946Z 03:02.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7398179Z 03:02.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7398416Z 03:02.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7398659Z 03:02.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7398898Z 03:02.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7399233Z 03:03.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7399472Z 03:03.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7399723Z 03:03.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7399962Z 03:03.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7400206Z 03:03.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7400439Z 03:03.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7400685Z 03:03.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7400928Z 03:03.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7401170Z 24:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7401409Z 25:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7401649Z 26:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7401893Z 26:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7402130Z 26:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7402366Z 26:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7402605Z 26:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7402846Z 26:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7403087Z 26:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7403501Z 26:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7403744Z 26:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7404062Z 27:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-10-10T01:29:29.7404389Z 30:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7404653Z 31:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7404898Z 32:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7405211Z 33:00.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-10-10T01:29:29.7405657Z 34:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:29.7405970Z 35:00.0 3D controller: NVIDIA Corporation AD104GL [L4] (rev a1) 2025-10-10T01:29:29.7406228Z + lsmod 2025-10-10T01:29:29.7438732Z Module Size Used by 2025-10-10T01:29:29.7439015Z nvidia_uvm 1925120 0 2025-10-10T01:29:29.7439347Z nvidia 14286848 1 nvidia_uvm 2025-10-10T01:29:29.7439616Z drm 602112 1 nvidia 2025-10-10T01:29:29.7439911Z drm_panel_orientation_quirks 32768 1 drm 2025-10-10T01:29:29.7440210Z backlight 24576 1 drm 2025-10-10T01:29:29.7440495Z i2c_core 110592 2 nvidia,drm 2025-10-10T01:29:29.7440760Z xt_conntrack 16384 1 2025-10-10T01:29:29.7440999Z nft_chain_nat 16384 3 2025-10-10T01:29:29.7441225Z xt_MASQUERADE 20480 1 2025-10-10T01:29:29.7441499Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-10-10T01:29:29.7441804Z nf_conntrack_netlink 57344 0 2025-10-10T01:29:29.7442179Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-10-10T01:29:29.7442597Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-10-10T01:29:29.7442875Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-10-10T01:29:29.7443134Z xfrm_user 57344 1 2025-10-10T01:29:29.7443384Z xfrm_algo 16384 1 xfrm_user 2025-10-10T01:29:29.7443622Z xt_addrtype 16384 2 2025-10-10T01:29:29.7443806Z nft_compat 20480 4 2025-10-10T01:29:29.7444027Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-10-10T01:29:29.7444329Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-10-10T01:29:29.7444604Z br_netfilter 36864 0 2025-10-10T01:29:29.7444808Z bridge 323584 1 br_netfilter 2025-10-10T01:29:29.7445027Z stp 16384 1 bridge 2025-10-10T01:29:29.7445240Z llc 16384 2 bridge,stp 2025-10-10T01:29:29.7445448Z overlay 167936 0 2025-10-10T01:29:29.7445626Z tls 139264 0 2025-10-10T01:29:29.7445808Z nls_ascii 16384 1 2025-10-10T01:29:29.7445987Z nls_cp437 20480 1 2025-10-10T01:29:29.7446164Z vfat 24576 1 2025-10-10T01:29:29.7446340Z fat 86016 1 vfat 2025-10-10T01:29:29.7446533Z sunrpc 700416 1 2025-10-10T01:29:29.7446712Z ena 184320 0 2025-10-10T01:29:29.7446903Z i8042 45056 0 2025-10-10T01:29:29.7447083Z serio 28672 3 i8042 2025-10-10T01:29:29.7447287Z ghash_clmulni_intel 16384 0 2025-10-10T01:29:29.7447481Z button 24576 0 2025-10-10T01:29:29.7447666Z sch_fq_codel 20480 9 2025-10-10T01:29:29.7447837Z fuse 184320 1 2025-10-10T01:29:29.7448012Z dm_mod 188416 0 2025-10-10T01:29:29.7448192Z configfs 57344 1 2025-10-10T01:29:29.7448372Z loop 36864 0 2025-10-10T01:29:29.7448551Z dmi_sysfs 20480 0 2025-10-10T01:29:29.7448736Z crc32_pclmul 16384 0 2025-10-10T01:29:29.7448923Z crc32c_intel 24576 0 2025-10-10T01:29:29.7449107Z efivarfs 24576 1 2025-10-10T01:29:29.7449297Z + modinfo nvidia 2025-10-10T01:29:29.7457608Z filename: /lib/modules/6.1.150-174.273.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-10-10T01:29:29.7458034Z import_ns: DMA_BUF 2025-10-10T01:29:29.7458423Z alias: char-major-195-* 2025-10-10T01:29:29.7458667Z version: 580.82.07 2025-10-10T01:29:29.7458911Z supported: external 2025-10-10T01:29:29.7459138Z license: Dual MIT/GPL 2025-10-10T01:29:29.7459401Z firmware: nvidia/580.82.07/gsp_tu10x.bin 2025-10-10T01:29:29.7459717Z firmware: nvidia/580.82.07/gsp_ga10x.bin 2025-10-10T01:29:29.7460004Z srcversion: BA7240A71DCF7DC6FE88C1D 2025-10-10T01:29:29.7460315Z alias: of:N*T*Cnvidia,tegra264-displayC* 2025-10-10T01:29:29.7460747Z alias: of:N*T*Cnvidia,tegra264-display 2025-10-10T01:29:29.7461078Z alias: of:N*T*Cnvidia,tegra234-displayC* 2025-10-10T01:29:29.7461404Z alias: of:N*T*Cnvidia,tegra234-display 2025-10-10T01:29:29.7461715Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-10-10T01:29:29.7462016Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-10-10T01:29:29.7462307Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-10-10T01:29:29.7462577Z depends: i2c-core,drm 2025-10-10T01:29:29.7462822Z retpoline: Y 2025-10-10T01:29:29.7463022Z name: nvidia 2025-10-10T01:29:29.7463367Z vermagic: 6.1.150-174.273.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-10-10T01:29:29.7463766Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-10-10T01:29:29.7464095Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-10-10T01:29:29.7464415Z parm: NVreg_ResmanDebugLevel:int 2025-10-10T01:29:29.7464646Z parm: NVreg_RmLogonRC:int 2025-10-10T01:29:29.7464869Z parm: NVreg_ModifyDeviceFiles:int 2025-10-10T01:29:29.7465102Z parm: NVreg_DeviceFileUID:int 2025-10-10T01:29:29.7465322Z parm: NVreg_DeviceFileGID:int 2025-10-10T01:29:29.7465553Z parm: NVreg_DeviceFileMode:int 2025-10-10T01:29:29.7465825Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-10-10T01:29:29.7466103Z parm: NVreg_UsePageAttributeTable:int 2025-10-10T01:29:29.7466349Z parm: NVreg_EnablePCIeGen3:int 2025-10-10T01:29:29.7466570Z parm: NVreg_EnableMSI:int 2025-10-10T01:29:29.7466791Z parm: NVreg_EnableStreamMemOPs:int 2025-10-10T01:29:29.7467052Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-10-10T01:29:29.7467347Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-10-10T01:29:29.7467623Z parm: NVreg_EnableS0ixPowerManagement:int 2025-10-10T01:29:29.7467934Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-10-10T01:29:29.7468244Z parm: NVreg_DynamicPowerManagement:int 2025-10-10T01:29:29.7468557Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-10-10T01:29:29.7468860Z parm: NVreg_EnableGpuFirmware:int 2025-10-10T01:29:29.7469106Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-10-10T01:29:29.7469375Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-10-10T01:29:29.7469653Z parm: NVreg_EnableUserNUMAManagement:int 2025-10-10T01:29:29.7469913Z parm: NVreg_MemoryPoolSize:int 2025-10-10T01:29:29.7470149Z parm: NVreg_KMallocHeapMaxSize:int 2025-10-10T01:29:29.7470387Z parm: NVreg_VMallocHeapMaxSize:int 2025-10-10T01:29:29.7470625Z parm: NVreg_IgnoreMMIOCheck:int 2025-10-10T01:29:29.7470864Z parm: NVreg_NvLinkDisable:int 2025-10-10T01:29:29.7471122Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-10-10T01:29:29.7471379Z parm: NVreg_RegisterPCIDriver:int 2025-10-10T01:29:29.7471639Z parm: NVreg_RegisterPlatformDeviceDriver:int 2025-10-10T01:29:29.7471899Z parm: NVreg_EnableResizableBar:int 2025-10-10T01:29:29.7472140Z parm: NVreg_EnableDbgBreakpoint:int 2025-10-10T01:29:29.7472387Z parm: NVreg_EnableNonblockingOpen:int 2025-10-10T01:29:29.7472639Z parm: NVreg_CoherentGPUMemoryMode:charp 2025-10-10T01:29:29.7472886Z parm: NVreg_RegistryDwords:charp 2025-10-10T01:29:29.7473132Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-10-10T01:29:29.7473465Z parm: NVreg_RmMsg:charp 2025-10-10T01:29:29.7473671Z parm: NVreg_GpuBlacklist:charp 2025-10-10T01:29:29.7473902Z parm: NVreg_TemporaryFilePath:charp 2025-10-10T01:29:29.7474141Z parm: NVreg_ExcludedGpus:charp 2025-10-10T01:29:29.7474368Z parm: NVreg_DmaRemapPeerMmio:int 2025-10-10T01:29:29.7474598Z parm: NVreg_RmNvlinkBandwidth:charp 2025-10-10T01:29:29.7474854Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-10-10T01:29:29.7475180Z parm: NVreg_ImexChannelCount:int 2025-10-10T01:29:29.7475424Z parm: NVreg_CreateImexChannel0:int 2025-10-10T01:29:29.7475669Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-10-10T01:29:29.7475914Z parm: rm_firmware_active:charp 2025-10-10T01:29:29.7476137Z + HAS_NVIDIA_DRIVER=0 2025-10-10T01:29:29.7476323Z ++ command -v nvidia-smi 2025-10-10T01:29:29.7476508Z + '[' -x /usr/bin/nvidia-smi ']' 2025-10-10T01:29:29.7476704Z + set +e 2025-10-10T01:29:29.7476935Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2025-10-10T01:29:31.3764236Z + INSTALLED_DRIVER_VERSION=580.82.07 2025-10-10T01:29:31.3764632Z + NVIDIA_SMI_STATUS=0 2025-10-10T01:29:31.3764876Z + '[' 0 -ne 0 ']' 2025-10-10T01:29:31.3765096Z + '[' 580.82.07 '!=' 580.82.07 ']' 2025-10-10T01:29:31.3765348Z + HAS_NVIDIA_DRIVER=1 2025-10-10T01:29:31.3765767Z + echo 'NVIDIA driver (580.82.07) has already been installed. Skipping NVIDIA driver installation' 2025-10-10T01:29:31.3766230Z + set -e 2025-10-10T01:29:31.3766460Z + '[' 1 -eq 0 ']' 2025-10-10T01:29:31.3766837Z NVIDIA driver (580.82.07) has already been installed. Skipping NVIDIA driver installation 2025-10-10T01:29:31.3767276Z + post_install_nvidia_driver_common 2025-10-10T01:29:31.3769817Z + sudo modprobe nvidia 2025-10-10T01:29:31.5289199Z + echo 'After installing NVIDIA driver' 2025-10-10T01:29:31.5289659Z + lspci 2025-10-10T01:29:31.5289940Z After installing NVIDIA driver 2025-10-10T01:29:31.5465051Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-10-10T01:29:31.5465587Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-10-10T01:29:31.5466117Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-10-10T01:29:31.5466616Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-10-10T01:29:31.5467062Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-10-10T01:29:31.5467495Z 01:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5467840Z 02:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5468163Z 03:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5468468Z 03:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5468760Z 03:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5469053Z 03:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5469362Z 03:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5469671Z 03:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5469959Z 03:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5470252Z 03:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5470546Z 03:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5470839Z 03:01.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5471144Z 03:01.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5471439Z 03:01.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5471738Z 03:01.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5472031Z 03:01.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5472320Z 03:01.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5472618Z 03:01.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5472913Z 03:02.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5473212Z 03:02.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5473820Z 03:02.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5474119Z 03:02.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5474415Z 03:02.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5474727Z 03:02.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5475029Z 03:02.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5475289Z 03:02.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5475531Z 03:03.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5475911Z 03:03.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5476171Z 03:03.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5476419Z 03:03.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5476659Z 03:03.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5476901Z 03:03.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5477167Z 03:03.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5477412Z 03:03.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5477658Z 24:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5477900Z 25:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5478142Z 26:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5478377Z 26:00.1 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5478621Z 26:00.2 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5478862Z 26:00.3 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5479197Z 26:00.4 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5479438Z 26:00.5 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5479681Z 26:00.6 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5479921Z 26:00.7 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5480172Z 26:01.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5480486Z 27:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-10-10T01:29:31.5480808Z 30:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5481047Z 31:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5481286Z 32:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5481599Z 33:00.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-10-10T01:29:31.5481928Z 34:00.0 PCI bridge: Amazon.com, Inc. Device 0200 2025-10-10T01:29:31.5482212Z 35:00.0 3D controller: NVIDIA Corporation AD104GL [L4] (rev a1) 2025-10-10T01:29:31.5482472Z + lsmod 2025-10-10T01:29:31.5502397Z Module Size Used by 2025-10-10T01:29:31.5502680Z nvidia_uvm 1925120 0 2025-10-10T01:29:31.5502931Z nvidia 14286848 1 nvidia_uvm 2025-10-10T01:29:31.5503201Z drm 602112 1 nvidia 2025-10-10T01:29:31.5503481Z drm_panel_orientation_quirks 32768 1 drm 2025-10-10T01:29:31.5503762Z backlight 24576 1 drm 2025-10-10T01:29:31.5504029Z i2c_core 110592 2 nvidia,drm 2025-10-10T01:29:31.5504297Z xt_conntrack 16384 1 2025-10-10T01:29:31.5504534Z nft_chain_nat 16384 3 2025-10-10T01:29:31.5504758Z xt_MASQUERADE 20480 1 2025-10-10T01:29:31.5505033Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-10-10T01:29:31.5505335Z nf_conntrack_netlink 57344 0 2025-10-10T01:29:31.5505720Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-10-10T01:29:31.5506132Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-10-10T01:29:31.5506411Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-10-10T01:29:31.5506672Z xfrm_user 57344 1 2025-10-10T01:29:31.5506915Z xfrm_algo 16384 1 xfrm_user 2025-10-10T01:29:31.5507175Z xt_addrtype 16384 2 2025-10-10T01:29:31.5507395Z nft_compat 20480 4 2025-10-10T01:29:31.5507670Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-10-10T01:29:31.5508049Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-10-10T01:29:31.5508592Z br_netfilter 36864 0 2025-10-10T01:29:31.5508848Z bridge 323584 1 br_netfilter 2025-10-10T01:29:31.5509114Z stp 16384 1 bridge 2025-10-10T01:29:31.5509383Z llc 16384 2 bridge,stp 2025-10-10T01:29:31.5509647Z overlay 167936 0 2025-10-10T01:29:31.5509871Z tls 139264 0 2025-10-10T01:29:31.5510084Z nls_ascii 16384 1 2025-10-10T01:29:31.5510323Z nls_cp437 20480 1 2025-10-10T01:29:31.5510678Z vfat 24576 1 2025-10-10T01:29:31.5510925Z fat 86016 1 vfat 2025-10-10T01:29:31.5511163Z sunrpc 700416 1 2025-10-10T01:29:31.5511389Z ena 184320 0 2025-10-10T01:29:31.5511605Z i8042 45056 0 2025-10-10T01:29:31.5511835Z serio 28672 3 i8042 2025-10-10T01:29:31.5512087Z ghash_clmulni_intel 16384 0 2025-10-10T01:29:31.5512339Z button 24576 0 2025-10-10T01:29:31.5512566Z sch_fq_codel 20480 9 2025-10-10T01:29:31.5512798Z fuse 184320 1 2025-10-10T01:29:31.5513012Z dm_mod 188416 0 2025-10-10T01:29:31.5513233Z configfs 57344 1 2025-10-10T01:29:31.5513452Z loop 36864 0 2025-10-10T01:29:31.5513667Z dmi_sysfs 20480 0 2025-10-10T01:29:31.5513882Z crc32_pclmul 16384 0 2025-10-10T01:29:31.5514104Z crc32c_intel 24576 0 2025-10-10T01:29:31.5514327Z efivarfs 24576 1 2025-10-10T01:29:31.5514551Z + modinfo nvidia 2025-10-10T01:29:31.5521894Z filename: /lib/modules/6.1.150-174.273.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-10-10T01:29:31.5522243Z import_ns: DMA_BUF 2025-10-10T01:29:31.5522429Z alias: char-major-195-* 2025-10-10T01:29:31.5522655Z version: 580.82.07 2025-10-10T01:29:31.5522838Z supported: external 2025-10-10T01:29:31.5523017Z license: Dual MIT/GPL 2025-10-10T01:29:31.5523251Z firmware: nvidia/580.82.07/gsp_tu10x.bin 2025-10-10T01:29:31.5523495Z firmware: nvidia/580.82.07/gsp_ga10x.bin 2025-10-10T01:29:31.5523734Z srcversion: BA7240A71DCF7DC6FE88C1D 2025-10-10T01:29:31.5523979Z alias: of:N*T*Cnvidia,tegra264-displayC* 2025-10-10T01:29:31.5524239Z alias: of:N*T*Cnvidia,tegra264-display 2025-10-10T01:29:31.5524490Z alias: of:N*T*Cnvidia,tegra234-displayC* 2025-10-10T01:29:31.5524739Z alias: of:N*T*Cnvidia,tegra234-display 2025-10-10T01:29:31.5525103Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-10-10T01:29:31.5525697Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-10-10T01:29:31.5526258Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-10-10T01:29:31.5526788Z depends: i2c-core,drm 2025-10-10T01:29:31.5527222Z retpoline: Y 2025-10-10T01:29:31.5527610Z name: nvidia 2025-10-10T01:29:31.5528276Z vermagic: 6.1.150-174.273.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-10-10T01:29:31.5529114Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-10-10T01:29:31.5529883Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-10-10T01:29:31.5530609Z parm: NVreg_ResmanDebugLevel:int 2025-10-10T01:29:31.5531142Z parm: NVreg_RmLogonRC:int 2025-10-10T01:29:31.5531660Z parm: NVreg_ModifyDeviceFiles:int 2025-10-10T01:29:31.5532203Z parm: NVreg_DeviceFileUID:int 2025-10-10T01:29:31.5532713Z parm: NVreg_DeviceFileGID:int 2025-10-10T01:29:31.5533236Z parm: NVreg_DeviceFileMode:int 2025-10-10T01:29:31.5533859Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-10-10T01:29:31.5534523Z parm: NVreg_UsePageAttributeTable:int 2025-10-10T01:29:31.5535075Z parm: NVreg_EnablePCIeGen3:int 2025-10-10T01:29:31.5535588Z parm: NVreg_EnableMSI:int 2025-10-10T01:29:31.5536028Z parm: NVreg_EnableStreamMemOPs:int 2025-10-10T01:29:31.5536422Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-10-10T01:29:31.5536724Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-10-10T01:29:31.5537004Z parm: NVreg_EnableS0ixPowerManagement:int 2025-10-10T01:29:31.5537305Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-10-10T01:29:31.5537612Z parm: NVreg_DynamicPowerManagement:int 2025-10-10T01:29:31.5537916Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-10-10T01:29:31.5538332Z parm: NVreg_EnableGpuFirmware:int 2025-10-10T01:29:31.5538588Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-10-10T01:29:31.5538859Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-10-10T01:29:31.5539125Z parm: NVreg_EnableUserNUMAManagement:int 2025-10-10T01:29:31.5539374Z parm: NVreg_MemoryPoolSize:int 2025-10-10T01:29:31.5539607Z parm: NVreg_KMallocHeapMaxSize:int 2025-10-10T01:29:31.5539846Z parm: NVreg_VMallocHeapMaxSize:int 2025-10-10T01:29:31.5540086Z parm: NVreg_IgnoreMMIOCheck:int 2025-10-10T01:29:31.5540307Z parm: NVreg_NvLinkDisable:int 2025-10-10T01:29:31.5540562Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-10-10T01:29:31.5540826Z parm: NVreg_RegisterPCIDriver:int 2025-10-10T01:29:31.5541086Z parm: NVreg_RegisterPlatformDeviceDriver:int 2025-10-10T01:29:31.5541352Z parm: NVreg_EnableResizableBar:int 2025-10-10T01:29:31.5541598Z parm: NVreg_EnableDbgBreakpoint:int 2025-10-10T01:29:31.5541850Z parm: NVreg_EnableNonblockingOpen:int 2025-10-10T01:29:31.5542104Z parm: NVreg_CoherentGPUMemoryMode:charp 2025-10-10T01:29:31.5542346Z parm: NVreg_RegistryDwords:charp 2025-10-10T01:29:31.5542594Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-10-10T01:29:31.5542837Z parm: NVreg_RmMsg:charp 2025-10-10T01:29:31.5543048Z parm: NVreg_GpuBlacklist:charp 2025-10-10T01:29:31.5543284Z parm: NVreg_TemporaryFilePath:charp 2025-10-10T01:29:31.5543520Z parm: NVreg_ExcludedGpus:charp 2025-10-10T01:29:31.5543748Z parm: NVreg_DmaRemapPeerMmio:int 2025-10-10T01:29:31.5543985Z parm: NVreg_RmNvlinkBandwidth:charp 2025-10-10T01:29:31.5544237Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-10-10T01:29:31.5544490Z parm: NVreg_ImexChannelCount:int 2025-10-10T01:29:31.5544724Z parm: NVreg_CreateImexChannel0:int 2025-10-10T01:29:31.5544986Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-10-10T01:29:31.5545227Z parm: rm_firmware_active:charp 2025-10-10T01:29:31.5545435Z + set +e 2025-10-10T01:29:31.5545578Z + nvidia-smi 2025-10-10T01:29:32.9822269Z Fri Oct 10 01:29:32 2025 2025-10-10T01:29:32.9823053Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:29:32.9824027Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-10-10T01:29:32.9824927Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:29:32.9825799Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-10-10T01:29:32.9826789Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-10-10T01:29:32.9827251Z | | | MIG M. | 2025-10-10T01:29:32.9827516Z |=========================================+========================+======================| 2025-10-10T01:29:32.9891237Z | 0 NVIDIA L4 Off | 00000000:35:00.0 Off | 0 | 2025-10-10T01:29:32.9891664Z | N/A 45C P0 30W / 72W | 0MiB / 23034MiB | 4% Default | 2025-10-10T01:29:32.9892023Z | | | N/A | 2025-10-10T01:29:32.9892384Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:29:32.9893031Z 2025-10-10T01:29:32.9893320Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:29:32.9893734Z | Processes: | 2025-10-10T01:29:32.9894127Z | GPU GI CI PID Type Process name GPU Memory | 2025-10-10T01:29:32.9894720Z | ID ID Usage | 2025-10-10T01:29:32.9895039Z |=========================================================================================| 2025-10-10T01:29:32.9896360Z | No running processes found | 2025-10-10T01:29:32.9896824Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:29:33.3118633Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-10-10T01:29:34.7343631Z NVIDIA L4 2025-10-10T01:29:34.9137244Z + NVIDIA_SMI_STATUS=0 2025-10-10T01:29:34.9137497Z + '[' 0 -eq 0 ']' 2025-10-10T01:29:34.9137692Z + echo 'INFO: Ignoring allowed status 0' 2025-10-10T01:29:34.9137921Z + set -e 2025-10-10T01:29:34.9138091Z INFO: Ignoring allowed status 0 2025-10-10T01:29:34.9146100Z == Installing nvidia container toolkit for amzn2023 == 2025-10-10T01:29:34.9150132Z + sudo yum install -y yum-utils 2025-10-10T01:29:35.3087822Z Last metadata expiration check: 0:07:57 ago on Fri Oct 10 01:21:38 2025. 2025-10-10T01:29:35.3303240Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed. 2025-10-10T01:29:35.3704650Z Dependencies resolved. 2025-10-10T01:29:35.3942624Z Nothing to do. 2025-10-10T01:29:35.3943209Z Complete! 2025-10-10T01:29:35.4942186Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2025-10-10T01:29:35.4942707Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-10-10T01:29:35.4943407Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-10-10T01:29:35.8294066Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-10-10T01:29:35.8757686Z + sudo yum install -y nvidia-container-toolkit-1.17.8 libnvidia-container-tools-1.17.8 libnvidia-container1-1.17.8 nvidia-container-toolkit-base-1.17.8 2025-10-10T01:29:36.3548098Z nvidia-container-toolkit 23 kB/s | 833 B 00:00 2025-10-10T01:29:36.3776553Z Package nvidia-container-toolkit-1.17.8-1.x86_64 is already installed. 2025-10-10T01:29:36.3782473Z Package libnvidia-container-tools-1.17.8-1.x86_64 is already installed. 2025-10-10T01:29:36.3786908Z Package libnvidia-container1-1.17.8-1.x86_64 is already installed. 2025-10-10T01:29:36.3793673Z Package nvidia-container-toolkit-base-1.17.8-1.x86_64 is already installed. 2025-10-10T01:29:36.4228748Z Dependencies resolved. 2025-10-10T01:29:36.4458805Z Nothing to do. 2025-10-10T01:29:36.4459157Z Complete! 2025-10-10T01:29:36.4999986Z + sudo systemctl restart docker 2025-10-10T01:30:32.0476870Z Fri Oct 10 01:30:32 2025 2025-10-10T01:30:32.0477304Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:30:32.0477784Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-10-10T01:30:32.0478237Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:30:32.0478739Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-10-10T01:30:32.0479376Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-10-10T01:30:32.0479784Z | | | MIG M. | 2025-10-10T01:30:32.0480094Z |=========================================+========================+======================| 2025-10-10T01:30:32.0552353Z | 0 NVIDIA L4 On | 00000000:35:00.0 Off | 0 | 2025-10-10T01:30:32.0552816Z | N/A 45C P0 30W / 72W | 0MiB / 23034MiB | 4% Default | 2025-10-10T01:30:32.0553173Z | | | N/A | 2025-10-10T01:30:32.0553534Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:30:32.0553970Z 2025-10-10T01:30:32.0554427Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:30:32.0554838Z | Processes: | 2025-10-10T01:30:32.0555259Z | GPU GI CI PID Type Process name GPU Memory | 2025-10-10T01:30:32.0555651Z | ID ID Usage | 2025-10-10T01:30:32.0555982Z |=========================================================================================| 2025-10-10T01:30:32.0557732Z | No running processes found | 2025-10-10T01:30:32.0558176Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:30:32.2179198Z Unable to find image 'public.ecr.aws/docker/library/python:3.13' locally 2025-10-10T01:30:32.4092906Z 3.13: Pulling from docker/library/python 2025-10-10T01:30:32.5007226Z cae3b572364a: Pulling fs layer 2025-10-10T01:30:32.5007595Z bd090f42c4b7: Pulling fs layer 2025-10-10T01:30:32.5007870Z f0c9d6d993ac: Pulling fs layer 2025-10-10T01:30:32.5008122Z a2ade626d67a: Pulling fs layer 2025-10-10T01:30:32.5008364Z 7f924d696c9c: Pulling fs layer 2025-10-10T01:30:32.5008592Z 12e6ee790ad5: Pulling fs layer 2025-10-10T01:30:32.5008845Z 54ea66483f67: Pulling fs layer 2025-10-10T01:30:32.5009070Z 12e6ee790ad5: Waiting 2025-10-10T01:30:32.5009283Z 54ea66483f67: Waiting 2025-10-10T01:30:32.5009497Z a2ade626d67a: Waiting 2025-10-10T01:30:32.5009711Z 7f924d696c9c: Waiting 2025-10-10T01:30:32.6087062Z bd090f42c4b7: Verifying Checksum 2025-10-10T01:30:32.6087348Z bd090f42c4b7: Download complete 2025-10-10T01:30:32.6971850Z cae3b572364a: Verifying Checksum 2025-10-10T01:30:32.6972180Z cae3b572364a: Download complete 2025-10-10T01:30:32.7779325Z 7f924d696c9c: Verifying Checksum 2025-10-10T01:30:32.7779793Z 7f924d696c9c: Download complete 2025-10-10T01:30:32.9119700Z 12e6ee790ad5: Verifying Checksum 2025-10-10T01:30:32.9120118Z 12e6ee790ad5: Download complete 2025-10-10T01:30:32.9510119Z f0c9d6d993ac: Verifying Checksum 2025-10-10T01:30:32.9510426Z f0c9d6d993ac: Download complete 2025-10-10T01:30:32.9562692Z 54ea66483f67: Download complete 2025-10-10T01:30:33.4863633Z a2ade626d67a: Verifying Checksum 2025-10-10T01:30:33.4864007Z a2ade626d67a: Download complete 2025-10-10T01:30:34.0232227Z cae3b572364a: Pull complete 2025-10-10T01:30:34.5603821Z bd090f42c4b7: Pull complete 2025-10-10T01:30:36.3606505Z f0c9d6d993ac: Pull complete 2025-10-10T01:30:41.6149969Z a2ade626d67a: Pull complete 2025-10-10T01:30:41.9691776Z 7f924d696c9c: Pull complete 2025-10-10T01:30:42.6251373Z 12e6ee790ad5: Pull complete 2025-10-10T01:30:42.6483680Z 54ea66483f67: Pull complete 2025-10-10T01:30:42.6620546Z Digest: sha256:4889af0e45f04b7c5dd741421a1280919499d38d3125d714b69fa86b23b1052a 2025-10-10T01:30:42.6662717Z Status: Downloaded newer image for public.ecr.aws/docker/library/python:3.13 2025-10-10T01:30:50.5535851Z Fri Oct 10 01:30:50 2025 2025-10-10T01:30:50.5536582Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:30:50.5537332Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-10-10T01:30:50.5537755Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:30:50.5538136Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-10-10T01:30:50.5538811Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-10-10T01:30:50.5539142Z | | | MIG M. | 2025-10-10T01:30:50.5539390Z |=========================================+========================+======================| 2025-10-10T01:30:50.5641783Z | 0 NVIDIA L4 On | 00000000:35:00.0 Off | 0 | 2025-10-10T01:30:50.5642500Z | N/A 43C P8 13W / 72W | 0MiB / 23034MiB | 0% Default | 2025-10-10T01:30:50.5642874Z | | | N/A | 2025-10-10T01:30:50.5643229Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T01:30:50.5645259Z 2025-10-10T01:30:50.5645468Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:30:50.5645906Z | Processes: | 2025-10-10T01:30:50.5646315Z | GPU GI CI PID Type Process name GPU Memory | 2025-10-10T01:30:50.5646690Z | ID ID Usage | 2025-10-10T01:30:50.5646994Z |=========================================================================================| 2025-10-10T01:30:50.5650200Z | No running processes found | 2025-10-10T01:30:50.5650586Z +-----------------------------------------------------------------------------------------+ 2025-10-10T01:30:51.9415506Z Command completed after 1 attempt(s). 2025-10-10T01:30:51.9513444Z Prepare all required actions 2025-10-10T01:30:51.9538038Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-10-10T01:30:51.9538308Z with: 2025-10-10T01:30:51.9538885Z github-token: *** 2025-10-10T01:30:51.9539092Z env: 2025-10-10T01:30:51.9539244Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:30:51.9539486Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:30:51.9539743Z ##[endgroup] 2025-10-10T01:30:51.9562005Z ##[group]Run set -eux 2025-10-10T01:30:51.9562201Z set -eux 2025-10-10T01:30:51.9562517Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-10-10T01:30:51.9575249Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:30:51.9575526Z env: 2025-10-10T01:30:51.9575700Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:30:51.9575952Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:30:51.9576380Z GITHUB_TOKEN: *** 2025-10-10T01:30:51.9576556Z ##[endgroup] 2025-10-10T01:30:51.9612588Z + python3 .github/scripts/get_workflow_job_id.py 18392306035 i-0f89836b6c30bbee2 2025-10-10T01:30:52.5636215Z Setting output job-id=52408504898 2025-10-10T01:30:52.5636818Z Setting output job-name=ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:30:52.5757444Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-10-10T01:30:52.5758032Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-10-10T01:30:52.5758724Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-10-10T01:30:52.5759431Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:30:52.5769302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:30:52.5769587Z env: 2025-10-10T01:30:52.5769746Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:30:52.5769991Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:30:52.5770248Z JOB_ID: 52408504898 2025-10-10T01:30:52.5770595Z JOB_NAME: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:30:52.5771177Z WORKFLOW_NAME: vllm-test 2025-10-10T01:30:52.5771389Z WORKFLOW_RUN_ID: 18392306035 2025-10-10T01:30:52.5771586Z MONITOR_LOG_INTERVAL: 5 2025-10-10T01:30:52.5771786Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-10-10T01:30:52.5771993Z ##[endgroup] 2025-10-10T01:30:52.8547588Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T01:30:53.1923881Z Collecting psutil==5.9.8 2025-10-10T01:30:53.2124142Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-10-10T01:30:53.2826408Z Collecting dataclasses_json==0.6.7 2025-10-10T01:30:53.2868632Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-10-10T01:30:53.3162248Z Collecting nvidia-ml-py==11.525.84 2025-10-10T01:30:53.3204223Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-10-10T01:30:53.3564113Z Collecting typing-inspect<1,>=0.4.0 2025-10-10T01:30:53.3770299Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-10-10T01:30:53.4776210Z Collecting marshmallow<4.0.0,>=3.18.0 2025-10-10T01:30:53.4843304Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-10-10T01:30:53.5371210Z Collecting packaging>=17.0 2025-10-10T01:30:53.5407094Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-10-10T01:30:53.5633616Z Collecting mypy-extensions>=0.3.0 2025-10-10T01:30:53.5673039Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-10-10T01:30:53.6110775Z Collecting typing-extensions>=3.7.4 2025-10-10T01:30:53.6173158Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-10-10T01:30:53.7063021Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-10-10T01:30:53.9561669Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-10-10T01:30:54.1243891Z Prepare all required actions 2025-10-10T01:30:54.1244253Z Getting action download info 2025-10-10T01:30:54.3883758Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-10-10T01:30:54.6091725Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-10-10T01:30:54.9029180Z ##[group]Run ./.github/actions/download-build-artifacts 2025-10-10T01:30:54.9029454Z with: 2025-10-10T01:30:54.9029640Z name: linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T01:30:54.9029871Z s3-bucket: gha-artifacts 2025-10-10T01:30:54.9030057Z env: 2025-10-10T01:30:54.9030225Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:30:54.9030475Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:30:54.9030724Z ##[endgroup] 2025-10-10T01:30:54.9058830Z ##[group]Run seemethere/download-artifact-s3@v4 2025-10-10T01:30:54.9059090Z with: 2025-10-10T01:30:54.9059313Z name: linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T01:30:54.9059550Z s3-bucket: gha-artifacts 2025-10-10T01:30:54.9059750Z region: us-east-1 2025-10-10T01:30:54.9059917Z env: 2025-10-10T01:30:54.9060083Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:30:54.9060324Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:30:54.9060580Z ##[endgroup] 2025-10-10T01:30:55.3406638Z (node:60331) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-10-10T01:30:55.3407116Z 2025-10-10T01:30:55.3407296Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-10-10T01:30:55.3407803Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-10-10T01:30:55.3408301Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-10-10T01:30:55.6553315Z Found 1 objects with prefix pytorch/pytorch/18392306035/linux-jammy-cuda12.8-py3.12-gcc11/ 2025-10-10T01:30:55.6554039Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-10-10T01:31:14.4983720Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-10-10T01:31:14.4989305Z Artifact download has finished successfully 2025-10-10T01:31:14.5249904Z ##[group]Run unzip -o artifacts.zip 2025-10-10T01:31:14.5250157Z unzip -o artifacts.zip 2025-10-10T01:31:14.5258053Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:14.5258327Z env: 2025-10-10T01:31:14.5258488Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:14.5258725Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:14.5258986Z ##[endgroup] 2025-10-10T01:31:14.5771028Z Archive: artifacts.zip 2025-10-10T01:31:14.5772229Z creating: dist/ 2025-10-10T01:31:17.0466895Z inflating: dist/torch-2.10.0a0+git344e636-cp312-cp312-linux_x86_64.whl 2025-10-10T01:31:17.0467380Z creating: dist/vision/ 2025-10-10T01:31:17.0593708Z inflating: dist/vision/torchvision-0.22.0a0+966da7e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:31:17.0594232Z creating: dist/audio/ 2025-10-10T01:31:17.0671184Z inflating: dist/audio/torchaudio-2.8.0a0+87ff22e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:31:17.0786886Z inflating: dist/.ninja_log 2025-10-10T01:31:17.0787204Z creating: dist/external/ 2025-10-10T01:31:17.0787474Z creating: dist/external/vllm/ 2025-10-10T01:31:17.0788201Z creating: dist/external/vllm/wheels/ 2025-10-10T01:31:17.0789022Z inflating: dist/external/vllm/wheels/build_summary.txt 2025-10-10T01:31:17.0789495Z creating: dist/external/vllm/wheels/flashinfer-python/ 2025-10-10T01:31:17.6951212Z inflating: dist/external/vllm/wheels/flashinfer-python/flashinfer_python-0.2.14.post1-cp39-abi3-linux_x86_64.whl 2025-10-10T01:31:17.6951880Z creating: dist/external/vllm/wheels/vllm/ 2025-10-10T01:31:19.9986556Z inflating: dist/external/vllm/wheels/vllm/vllm-0.11.0rc2.dev157+g0ad9951c4.d20251010-cp38-abi3-linux_x86_64.whl 2025-10-10T01:31:19.9987718Z creating: dist/external/vllm/wheels/xformers/ 2025-10-10T01:31:21.5626702Z inflating: dist/external/vllm/wheels/xformers/xformers-0.0.33+5d4b92a5.d20251010-cp39-abi3-linux_x86_64.whl 2025-10-10T01:31:21.5627392Z creating: build/custom_test_artifacts/ 2025-10-10T01:31:21.5627776Z creating: build/custom_test_artifacts/custom-op-build/ 2025-10-10T01:31:21.5628235Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-10-10T01:31:21.5628756Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-10-10T01:31:21.5636364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-10-10T01:31:21.5636967Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-10-10T01:31:21.5637588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-10-10T01:31:21.5638230Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-10-10T01:31:21.5638877Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-10-10T01:31:21.5641823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-10-10T01:31:21.5643086Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-10-10T01:31:21.5644103Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-10-10T01:31:21.5644749Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-10-10T01:31:21.5645373Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-10-10T01:31:21.5647951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-10-10T01:31:21.5649578Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-10-10T01:31:21.5650662Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-10-10T01:31:21.5652391Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-10-10T01:31:21.5654316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-10-10T01:31:21.5654899Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-10-10T01:31:21.5655405Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-10-10T01:31:21.5707803Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-10-10T01:31:21.5760472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-10-10T01:31:21.5761453Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-10-10T01:31:21.5817569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-10-10T01:31:21.5818489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-10-10T01:31:21.5838281Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-10-10T01:31:21.5839361Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-10-10T01:31:21.5840140Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-10-10T01:31:21.5840864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-10-10T01:31:21.5841880Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-10-10T01:31:21.5842616Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-10-10T01:31:21.5843359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-10-10T01:31:21.5844023Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-10-10T01:31:21.5844665Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-10-10T01:31:21.5845310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-10-10T01:31:21.5845951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-10-10T01:31:21.5846575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-10-10T01:31:21.5847204Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-10-10T01:31:21.5898212Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-10-10T01:31:21.5899098Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-10-10T01:31:21.5964175Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-10-10T01:31:21.5964905Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-10-10T01:31:21.5965470Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-10-10T01:31:21.5966056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-10-10T01:31:21.5966930Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-10-10T01:31:21.5967606Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-10-10T01:31:21.5968476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-10-10T01:31:21.5969224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-10-10T01:31:21.5970091Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-10-10T01:31:21.5970942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-10-10T01:31:21.5972203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-10-10T01:31:21.5972995Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-10-10T01:31:21.5973864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-10-10T01:31:21.5974864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-10-10T01:31:21.5994117Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-10-10T01:31:21.6169689Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-10-10T01:31:21.6170345Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-10-10T01:31:21.6171060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-10-10T01:31:21.6171879Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-10-10T01:31:21.6172770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-10-10T01:31:21.6173890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-10-10T01:31:21.6174643Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-10-10T01:31:21.6175676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-10-10T01:31:21.6176417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-10-10T01:31:21.6177244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-10-10T01:31:21.6178254Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-10-10T01:31:21.6197089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-10-10T01:31:21.6267018Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-10-10T01:31:21.6267860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-10-10T01:31:21.6268787Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-10-10T01:31:21.6269455Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-10-10T01:31:21.6270642Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-10-10T01:31:21.6272389Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-10-10T01:31:21.6272976Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2025-10-10T01:31:21.6275822Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-10-10T01:31:21.6276722Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-10-10T01:31:21.6277582Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-10-10T01:31:21.6428005Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-10-10T01:31:21.6476652Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-10-10T01:31:21.6477120Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-10-10T01:31:21.6477549Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-10-10T01:31:21.6478214Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-10-10T01:31:21.6485772Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-10-10T01:31:21.6486380Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-10-10T01:31:21.6487095Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-10-10T01:31:21.6487710Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-10-10T01:31:21.6488399Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-10-10T01:31:21.6491094Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-10-10T01:31:21.6492543Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-10-10T01:31:21.6493550Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-10-10T01:31:21.6494180Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-10-10T01:31:21.6494789Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-10-10T01:31:21.6497571Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-10-10T01:31:21.6499112Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-10-10T01:31:21.6500328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-10-10T01:31:21.6502005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-10-10T01:31:21.6503819Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-10-10T01:31:21.6504380Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-10-10T01:31:21.6504875Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-10-10T01:31:21.6556917Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-10-10T01:31:21.6609783Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-10-10T01:31:21.6610715Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-10-10T01:31:21.6666768Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-10-10T01:31:21.6667672Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-10-10T01:31:21.6669026Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-10-10T01:31:21.6670090Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-10-10T01:31:21.6671051Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-10-10T01:31:21.6671998Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-10-10T01:31:21.6673091Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-10-10T01:31:21.6674175Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-10-10T01:31:21.6675550Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-10-10T01:31:21.6676276Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-10-10T01:31:21.6677126Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-10-10T01:31:21.6678071Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-10-10T01:31:21.6679181Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-10-10T01:31:21.6681047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-10-10T01:31:21.6683448Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-10-10T01:31:21.6747974Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-10-10T01:31:21.6748814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-10-10T01:31:21.6813733Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-10-10T01:31:21.6814416Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-10-10T01:31:21.6814941Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-10-10T01:31:21.6815524Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-10-10T01:31:21.6816104Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-10-10T01:31:21.6817403Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-10-10T01:31:21.6818183Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-10-10T01:31:21.6818932Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-10-10T01:31:21.6819698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-10-10T01:31:21.6820623Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-10-10T01:31:21.6821691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-10-10T01:31:21.6822508Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-10-10T01:31:21.6823288Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-10-10T01:31:21.6824362Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-10-10T01:31:21.6843201Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-10-10T01:31:21.6899292Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-10-10T01:31:21.6900164Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-10-10T01:31:21.6900905Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-10-10T01:31:21.6901688Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-10-10T01:31:21.6902810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-10-10T01:31:21.6904587Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-10-10T01:31:21.6905378Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2025-10-10T01:31:21.6907845Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-10-10T01:31:21.6908781Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-10-10T01:31:21.6909681Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-10-10T01:31:21.6944198Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-10-10T01:31:21.6944660Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-10-10T01:31:21.6945137Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-10-10T01:31:21.6945751Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-10-10T01:31:21.6953274Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-10-10T01:31:21.6953921Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-10-10T01:31:21.6954551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-10-10T01:31:21.6955228Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-10-10T01:31:21.6955812Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-10-10T01:31:21.6958345Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-10-10T01:31:21.6959995Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-10-10T01:31:21.6960775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-10-10T01:31:21.6961324Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-10-10T01:31:21.6962207Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-10-10T01:31:21.6964831Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-10-10T01:31:21.6966026Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-10-10T01:31:21.6967201Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-10-10T01:31:21.6968958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-10-10T01:31:21.6970730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-10-10T01:31:21.6971347Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-10-10T01:31:21.6971882Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-10-10T01:31:21.7024589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-10-10T01:31:21.7077408Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-10-10T01:31:21.7078365Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-10-10T01:31:21.7134975Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-10-10T01:31:21.7135919Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-10-10T01:31:21.7137192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-10-10T01:31:21.7138281Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-10-10T01:31:21.7139538Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-10-10T01:31:21.7140481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-10-10T01:31:21.7141469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-10-10T01:31:21.7142535Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-10-10T01:31:21.7143830Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-10-10T01:31:21.7145143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-10-10T01:31:21.7145929Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-10-10T01:31:21.7146971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-10-10T01:31:21.7148017Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-10-10T01:31:21.7149073Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-10-10T01:31:21.7151682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-10-10T01:31:21.7216932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-10-10T01:31:21.7217723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-10-10T01:31:21.7282714Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-10-10T01:31:21.7283465Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-10-10T01:31:21.7284053Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-10-10T01:31:21.7284648Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-10-10T01:31:21.7285333Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-10-10T01:31:21.7286064Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-10-10T01:31:21.7286877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-10-10T01:31:21.7287725Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-10-10T01:31:21.7288617Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-10-10T01:31:21.7289427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-10-10T01:31:21.7290655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-10-10T01:31:21.7291487Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-10-10T01:31:21.7292249Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-10-10T01:31:21.7293305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-10-10T01:31:21.7298200Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-10-10T01:31:21.7402595Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-10-10T01:31:21.7403585Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-10-10T01:31:21.7404340Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-10-10T01:31:21.7405204Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-10-10T01:31:21.7405924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-10-10T01:31:21.7406792Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-10-10T01:31:21.7407807Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-10-10T01:31:21.7408809Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-10-10T01:31:21.7409737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-10-10T01:31:21.7410652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-10-10T01:31:21.7411596Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-10-10T01:31:21.7430334Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-10-10T01:31:21.7478730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-10-10T01:31:21.7479923Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-10-10T01:31:21.7481255Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-10-10T01:31:21.7482344Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-10-10T01:31:21.7483375Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-10-10T01:31:21.7484774Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-10-10T01:31:21.7485813Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2025-10-10T01:31:21.7488662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-10-10T01:31:21.7489570Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-10-10T01:31:21.7490425Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-10-10T01:31:21.7579562Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-10-10T01:31:21.7613940Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-10-10T01:31:21.7614550Z creating: build/lib/ 2025-10-10T01:31:21.7686184Z inflating: build/lib/libprotobuf-lite.a 2025-10-10T01:31:21.8070126Z inflating: build/lib/libprotobuf.a 2025-10-10T01:31:21.8078825Z inflating: build/lib/libpthreadpool.a 2025-10-10T01:31:21.8086417Z inflating: build/lib/libcpuinfo.a 2025-10-10T01:31:21.8093454Z inflating: build/lib/libcpuinfo_internals.a 2025-10-10T01:31:21.8094382Z inflating: build/lib/libclog.a 2025-10-10T01:31:21.8096827Z inflating: build/lib/libnnpack_reference_layers.a 2025-10-10T01:31:21.8527127Z inflating: build/lib/libprotoc.a 2025-10-10T01:31:21.8543846Z inflating: build/lib/libpytorch_qnnpack.a 2025-10-10T01:31:21.8559728Z inflating: build/lib/libnnpack.a 2025-10-10T01:31:21.8721242Z inflating: build/lib/libmicrokernels-prod.a 2025-10-10T01:31:21.9478782Z inflating: build/lib/libmicrokernels-all.a 2025-10-10T01:31:21.9544828Z inflating: build/lib/libbenchmark.a 2025-10-10T01:31:21.9545749Z inflating: build/lib/libbenchmark_main.a 2025-10-10T01:31:21.9607353Z inflating: build/lib/libgtest.a 2025-10-10T01:31:21.9622651Z inflating: build/lib/libgmock.a 2025-10-10T01:31:21.9623484Z inflating: build/lib/libgmock_main.a 2025-10-10T01:31:21.9624567Z inflating: build/lib/libgtest_main.a 2025-10-10T01:31:21.9703807Z inflating: build/lib/libXNNPACK.a 2025-10-10T01:31:21.9704671Z inflating: build/lib/libjitprofiling.a 2025-10-10T01:31:21.9712283Z inflating: build/lib/libittnotify.a 2025-10-10T01:31:21.9769248Z inflating: build/lib/libasmjit.a 2025-10-10T01:31:22.0794184Z inflating: build/lib/libfbgemm.a 2025-10-10T01:31:22.0821240Z inflating: build/lib/libtensorpipe_uv.a 2025-10-10T01:31:22.1301458Z inflating: build/lib/libtensorpipe.a 2025-10-10T01:31:22.1518267Z inflating: build/lib/libtensorpipe_cuda.a 2025-10-10T01:31:22.1635447Z inflating: build/lib/libgloo.a 2025-10-10T01:31:22.1676722Z inflating: build/lib/libonnx_proto.a 2025-10-10T01:31:22.2299600Z inflating: build/lib/libonnx.a 2025-10-10T01:31:22.2684045Z inflating: build/lib/libgloo_cuda.a 2025-10-10T01:31:22.2701551Z inflating: build/lib/libfmt.a 2025-10-10T01:31:23.1563463Z inflating: build/lib/libdnnl.a 2025-10-10T01:31:23.1975253Z inflating: build/lib/libkineto.a 2025-10-10T01:31:23.1976713Z inflating: build/lib/libtorch_global_deps.so 2025-10-10T01:31:23.2076494Z inflating: build/lib/libc10.so 2025-10-10T01:31:23.2078409Z inflating: build/lib/libcaffe2_nvrtc.so 2025-10-10T01:31:23.2130427Z inflating: build/lib/libc10_cuda.so 2025-10-10T01:31:25.8228477Z inflating: build/lib/libtorch_cpu.so 2025-10-10T01:31:25.8909868Z inflating: build/lib/libtorch_nvshmem.so 2025-10-10T01:31:25.8913907Z inflating: build/lib/libshm.so 2025-10-10T01:31:29.9028235Z inflating: build/lib/libtorch_cuda.so 2025-10-10T01:31:29.9029251Z inflating: build/lib/libtorch.so 2025-10-10T01:31:29.9032145Z inflating: build/lib/libc10d_cuda_test.so 2025-10-10T01:31:29.9076597Z inflating: build/lib/libtorch_cuda_linalg.so 2025-10-10T01:31:29.9092941Z inflating: build/lib/libjitbackend_test.so 2025-10-10T01:31:29.9155344Z inflating: build/lib/libtorchbind_test.so 2025-10-10T01:31:29.9176684Z inflating: build/lib/libbackend_with_compiler.so 2025-10-10T01:31:29.9200226Z inflating: build/lib/libaoti_custom_ops.so 2025-10-10T01:31:30.1174933Z inflating: build/lib/libtorch_python.so 2025-10-10T01:31:30.1206634Z inflating: build/lib/libnnapi_backend.so 2025-10-10T01:31:30.1207079Z creating: build/bin/ 2025-10-10T01:31:30.1602676Z inflating: build/bin/protoc-3.13.0.0 2025-10-10T01:31:30.1996603Z inflating: build/bin/protoc 2025-10-10T01:31:30.2044686Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-10-10T01:31:30.2094273Z inflating: build/bin/c10_DeviceGuard_test 2025-10-10T01:31:30.2151902Z inflating: build/bin/c10_DispatchKeySet_test 2025-10-10T01:31:30.2201876Z inflating: build/bin/c10_Device_test 2025-10-10T01:31:30.2256670Z inflating: build/bin/c10_SymInt_test 2025-10-10T01:31:30.2304355Z inflating: build/bin/c10_StreamGuard_test 2025-10-10T01:31:30.2358094Z inflating: build/bin/c10_SizesAndStrides_test 2025-10-10T01:31:30.2409652Z inflating: build/bin/c10_AllocatorConfig_test 2025-10-10T01:31:30.2458651Z inflating: build/bin/c10_Half_test 2025-10-10T01:31:30.2506906Z inflating: build/bin/c10_DeadlockDetection_test 2025-10-10T01:31:30.2554297Z inflating: build/bin/c10_ConstexprCrc_test 2025-10-10T01:31:30.2607114Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-10-10T01:31:30.2658208Z inflating: build/bin/c10_Bitset_test 2025-10-10T01:31:30.2706943Z inflating: build/bin/c10_generic_math_test 2025-10-10T01:31:30.2755142Z inflating: build/bin/c10_Synchronized_test 2025-10-10T01:31:30.2805285Z inflating: build/bin/c10_TypeIndex_test 2025-10-10T01:31:30.2858833Z inflating: build/bin/c10_LeftRight_test 2025-10-10T01:31:30.2929774Z inflating: build/bin/c10_optional_test 2025-10-10T01:31:30.2980909Z inflating: build/bin/c10_NetworkFlow_test 2025-10-10T01:31:30.3035489Z inflating: build/bin/c10_Enumerate_test 2025-10-10T01:31:30.3088437Z inflating: build/bin/c10_complex_test 2025-10-10T01:31:30.3141808Z inflating: build/bin/c10_ThreadLocal_test 2025-10-10T01:31:30.3189588Z inflating: build/bin/c10_error_test 2025-10-10T01:31:30.3239530Z inflating: build/bin/c10_accumulate_test 2025-10-10T01:31:30.3290766Z inflating: build/bin/c10_lazy_test 2025-10-10T01:31:30.3349802Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-10-10T01:31:30.3397886Z inflating: build/bin/c10_Semaphore_test 2025-10-10T01:31:30.3452575Z inflating: build/bin/c10_logging_test 2025-10-10T01:31:30.3501756Z inflating: build/bin/c10_ArrayRef_test 2025-10-10T01:31:30.3554961Z inflating: build/bin/c10_Metaprogramming_test 2025-10-10T01:31:30.3606355Z inflating: build/bin/c10_exception_test 2025-10-10T01:31:30.3657366Z inflating: build/bin/c10_IntrusiveList_test 2025-10-10T01:31:30.3724723Z inflating: build/bin/c10_cow_test 2025-10-10T01:31:30.3772166Z inflating: build/bin/c10_TypeTraits_test 2025-10-10T01:31:30.3821400Z inflating: build/bin/c10_irange_test 2025-10-10T01:31:30.3875360Z inflating: build/bin/c10_bfloat16_test 2025-10-10T01:31:30.3928066Z inflating: build/bin/c10_Scalar_test 2025-10-10T01:31:30.3982014Z inflating: build/bin/c10_string_util_test 2025-10-10T01:31:30.4030636Z inflating: build/bin/c10_flags_test 2025-10-10T01:31:30.4081814Z inflating: build/bin/c10_registry_test 2025-10-10T01:31:30.4132194Z inflating: build/bin/c10_ssize_test 2025-10-10T01:31:30.4185496Z inflating: build/bin/c10_InlineStreamGuard_test 2025-10-10T01:31:30.4233135Z inflating: build/bin/c10_string_view_test 2025-10-10T01:31:30.4281583Z inflating: build/bin/c10_tempfile_test 2025-10-10T01:31:30.4330929Z inflating: build/bin/c10_TypeList_test 2025-10-10T01:31:30.4373459Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-10-10T01:31:30.4427741Z inflating: build/bin/c10_complex_math_test 2025-10-10T01:31:30.4476267Z inflating: build/bin/c10_bit_cast_test 2025-10-10T01:31:30.4530299Z inflating: build/bin/c10_typeid_test 2025-10-10T01:31:30.4682624Z inflating: build/bin/c10_intrusive_ptr_test 2025-10-10T01:31:30.4824182Z inflating: build/bin/c10_small_vector_test 2025-10-10T01:31:30.4871790Z inflating: build/bin/c10_cuda_CUDATest 2025-10-10T01:31:30.5416199Z inflating: build/bin/vec_test_all_types_AVX2 2025-10-10T01:31:30.5943917Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-10-10T01:31:30.6482874Z inflating: build/bin/vec_test_all_types_AVX512 2025-10-10T01:31:30.6533432Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2025-10-10T01:31:30.6584473Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2025-10-10T01:31:30.6636006Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2025-10-10T01:31:30.6687027Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2025-10-10T01:31:30.6737897Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2025-10-10T01:31:30.6788415Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2025-10-10T01:31:30.6839436Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2025-10-10T01:31:30.6893810Z inflating: build/bin/TCPStoreTest 2025-10-10T01:31:30.6945188Z inflating: build/bin/FileStoreTest 2025-10-10T01:31:30.6996988Z inflating: build/bin/HashStoreTest 2025-10-10T01:31:30.7047567Z inflating: build/bin/BackoffTest 2025-10-10T01:31:30.7060004Z inflating: build/bin/ProcessGroupMPITest 2025-10-10T01:31:30.7063585Z inflating: build/bin/torch_shm_manager 2025-10-10T01:31:30.7066583Z inflating: build/bin/example_allreduce 2025-10-10T01:31:30.7124956Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2025-10-10T01:31:30.7179437Z inflating: build/bin/ProcessGroupGlooAsyncTest 2025-10-10T01:31:30.7240240Z inflating: build/bin/ProcessGroupNCCLTest 2025-10-10T01:31:30.7289340Z inflating: build/bin/dlconvertor_test 2025-10-10T01:31:30.7344321Z inflating: build/bin/extension_backend_test 2025-10-10T01:31:30.7407374Z inflating: build/bin/ProcessGroupGlooTest 2025-10-10T01:31:30.7477102Z inflating: build/bin/Dict_test 2025-10-10T01:31:30.7531145Z inflating: build/bin/broadcast_test 2025-10-10T01:31:30.7587659Z inflating: build/bin/atest 2025-10-10T01:31:30.7641157Z inflating: build/bin/half_test 2025-10-10T01:31:30.7691664Z inflating: build/bin/Dimname_test 2025-10-10T01:31:30.7746222Z inflating: build/bin/NamedTensor_test 2025-10-10T01:31:30.7808492Z inflating: build/bin/MaybeOwned_test 2025-10-10T01:31:30.7856552Z inflating: build/bin/lazy_tensor_test 2025-10-10T01:31:30.7909224Z inflating: build/bin/test_parallel 2025-10-10T01:31:30.7960418Z inflating: build/bin/cpu_profiling_allocator_test 2025-10-10T01:31:30.8021545Z inflating: build/bin/basic 2025-10-10T01:31:30.8070671Z inflating: build/bin/operators_test 2025-10-10T01:31:30.8156927Z inflating: build/bin/cpu_rng_test 2025-10-10T01:31:30.8268892Z inflating: build/bin/kernel_function_legacy_test 2025-10-10T01:31:30.8319165Z inflating: build/bin/stride_properties_test 2025-10-10T01:31:30.8368993Z inflating: build/bin/packedtensoraccessor_test 2025-10-10T01:31:30.8424713Z inflating: build/bin/cpu_generator_test 2025-10-10T01:31:30.8498974Z inflating: build/bin/tensor_iterator_test 2025-10-10T01:31:30.8553149Z inflating: build/bin/scalar_tensor_test 2025-10-10T01:31:30.8605602Z inflating: build/bin/math_kernel_test 2025-10-10T01:31:30.8661973Z inflating: build/bin/scalar_test 2025-10-10T01:31:30.8710474Z inflating: build/bin/reduce_ops_test 2025-10-10T01:31:30.8808961Z inflating: build/bin/List_test 2025-10-10T01:31:30.8873051Z inflating: build/bin/pow_test 2025-10-10T01:31:30.8921635Z inflating: build/bin/verify_api_visibility 2025-10-10T01:31:30.8970156Z inflating: build/bin/cuda_exchange_device_test 2025-10-10T01:31:30.9036376Z inflating: build/bin/legacy_vmap_test 2025-10-10T01:31:30.9085876Z inflating: build/bin/weakref_test 2025-10-10T01:31:30.9138761Z inflating: build/bin/type_ptr_test 2025-10-10T01:31:30.9190129Z inflating: build/bin/cuda_apply_test 2025-10-10T01:31:30.9239884Z inflating: build/bin/StorageUtils_test 2025-10-10T01:31:30.9303319Z inflating: build/bin/inline_container_test 2025-10-10T01:31:30.9352582Z inflating: build/bin/reportMemoryUsage_test 2025-10-10T01:31:30.9409834Z inflating: build/bin/apply_utils_test 2025-10-10T01:31:30.9467010Z inflating: build/bin/cuda_complex_test 2025-10-10T01:31:30.9517673Z inflating: build/bin/undefined_tensor_test 2025-10-10T01:31:30.9798119Z inflating: build/bin/op_registration_test 2025-10-10T01:31:30.9846956Z inflating: build/bin/operator_name_test 2025-10-10T01:31:30.9899510Z inflating: build/bin/memory_format_test 2025-10-10T01:31:30.9990309Z inflating: build/bin/ivalue_test 2025-10-10T01:31:31.0040694Z inflating: build/bin/CppSignature_test 2025-10-10T01:31:31.0093124Z inflating: build/bin/memory_overlapping_test 2025-10-10T01:31:31.0146968Z inflating: build/bin/native_test 2025-10-10T01:31:31.0202027Z inflating: build/bin/quantized_test 2025-10-10T01:31:31.0264729Z inflating: build/bin/KernelFunction_test 2025-10-10T01:31:31.0312191Z inflating: build/bin/op_allowlist_test 2025-10-10T01:31:31.0380454Z inflating: build/bin/cuda_complex_math_test 2025-10-10T01:31:31.0439277Z inflating: build/bin/cuda_atomic_ops_test 2025-10-10T01:31:31.0497221Z inflating: build/bin/kernel_stackbased_test 2025-10-10T01:31:31.0550276Z inflating: build/bin/backend_fallback_test 2025-10-10T01:31:31.0599552Z inflating: build/bin/cpu_allocator_test 2025-10-10T01:31:31.0653421Z inflating: build/bin/cuda_caching_host_allocator_test 2025-10-10T01:31:31.0710123Z inflating: build/bin/IListRef_test 2025-10-10T01:31:31.0772254Z inflating: build/bin/cuda_cub_test 2025-10-10T01:31:31.0828908Z inflating: build/bin/type_test 2025-10-10T01:31:31.0878515Z inflating: build/bin/xla_tensor_test 2025-10-10T01:31:31.0928328Z inflating: build/bin/wrapdim_test 2025-10-10T01:31:31.0986390Z inflating: build/bin/cuda_stream_test 2025-10-10T01:31:31.1036197Z inflating: build/bin/cuda_half_test 2025-10-10T01:31:31.1117115Z inflating: build/bin/cuda_distributions_test 2025-10-10T01:31:31.1165875Z inflating: build/bin/cuda_allocatorTraceTracker_test 2025-10-10T01:31:31.1215836Z inflating: build/bin/cuda_integer_divider_test 2025-10-10T01:31:31.1263897Z inflating: build/bin/cuda_optional_test 2025-10-10T01:31:31.1314698Z inflating: build/bin/cuda_packedtensoraccessor_test 2025-10-10T01:31:31.1363397Z inflating: build/bin/thread_init_test 2025-10-10T01:31:31.1412194Z inflating: build/bin/cuda_device_test 2025-10-10T01:31:31.1464353Z inflating: build/bin/mobile_memory_cleanup 2025-10-10T01:31:31.1514770Z inflating: build/bin/cuda_dlconvertor_test 2025-10-10T01:31:31.1603368Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-10-10T01:31:31.1719993Z inflating: build/bin/kernel_lambda_legacy_test 2025-10-10T01:31:31.1771664Z inflating: build/bin/cuda_allocator_test 2025-10-10T01:31:31.1836364Z inflating: build/bin/test_cpp_rpc 2025-10-10T01:31:31.1926572Z inflating: build/bin/kernel_function_test 2025-10-10T01:31:31.1979312Z inflating: build/bin/test_dist_autograd 2025-10-10T01:31:31.2031526Z inflating: build/bin/cuda_vectorized_test 2025-10-10T01:31:31.2082849Z inflating: build/bin/cuda_reportMemoryUsage_test 2025-10-10T01:31:31.2085228Z inflating: build/bin/parallel_benchmark 2025-10-10T01:31:31.2179930Z inflating: build/bin/kernel_lambda_test 2025-10-10T01:31:31.2491154Z inflating: build/bin/test_lazy 2025-10-10T01:31:31.2539665Z inflating: build/bin/cuda_cudnn_test 2025-10-10T01:31:31.2594275Z inflating: build/bin/cuda_generator_test 2025-10-10T01:31:31.3631432Z inflating: build/bin/test_api 2025-10-10T01:31:31.4611929Z inflating: build/bin/test_jit 2025-10-10T01:31:31.4612412Z creating: .additional_ci_files/ 2025-10-10T01:31:31.4673855Z inflating: .additional_ci_files/test-times.json 2025-10-10T01:31:31.4905101Z inflating: .additional_ci_files/test-class-times.json 2025-10-10T01:31:31.4952072Z ##[group]Run rm artifacts.zip 2025-10-10T01:31:31.4952313Z rm artifacts.zip 2025-10-10T01:31:31.4964817Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:31.4965105Z env: 2025-10-10T01:31:31.4965262Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:31.4965513Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:31.4965773Z ##[endgroup] 2025-10-10T01:31:31.6864805Z ##[group]Run df -H 2025-10-10T01:31:31.6865013Z df -H 2025-10-10T01:31:31.6872361Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:31.6872634Z env: 2025-10-10T01:31:31.6872803Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:31.6873048Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:31.6873312Z ##[endgroup] 2025-10-10T01:31:31.6922342Z Filesystem Size Used Avail Use% Mounted on 2025-10-10T01:31:31.6922718Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-10-10T01:31:31.6923030Z tmpfs 33G 0 33G 0% /dev/shm 2025-10-10T01:31:31.6923323Z tmpfs 13G 771k 13G 1% /run 2025-10-10T01:31:31.6923607Z /dev/nvme0n1p1 161G 56G 106G 35% / 2025-10-10T01:31:31.6923907Z tmpfs 33G 13k 33G 1% /tmp 2025-10-10T01:31:31.6924210Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-10-10T01:31:31.6924531Z tmpfs 6.5G 0 6.5G 0% /run/user/0 2025-10-10T01:31:31.6954131Z Prepare all required actions 2025-10-10T01:31:31.6954783Z Getting action download info 2025-10-10T01:31:31.8300866Z ##[group]Run ./.github/actions/download-td-artifacts 2025-10-10T01:31:31.8301123Z with: 2025-10-10T01:31:31.8301283Z env: 2025-10-10T01:31:31.8301440Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:31.8301683Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:31.8301934Z ##[endgroup] 2025-10-10T01:31:31.8348546Z ##[group]Run seemethere/download-artifact-s3@v4 2025-10-10T01:31:31.8348798Z with: 2025-10-10T01:31:31.8348947Z name: td_results 2025-10-10T01:31:31.8349128Z s3-bucket: gha-artifacts 2025-10-10T01:31:31.8349320Z region: us-east-1 2025-10-10T01:31:31.8349480Z env: 2025-10-10T01:31:31.8349626Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:31.8349870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:31.8350125Z ##[endgroup] 2025-10-10T01:31:32.2710014Z (node:60366) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-10-10T01:31:32.2710405Z 2025-10-10T01:31:32.2710556Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-10-10T01:31:32.2710943Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-10-10T01:31:32.2711343Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-10-10T01:31:32.3815736Z Found 0 objects with prefix pytorch/pytorch/18392306035/td_results/ 2025-10-10T01:31:32.3821539Z Artifact download has finished successfully 2025-10-10T01:31:32.4078058Z ##[group]Run mkdir -p .additional_ci_files 2025-10-10T01:31:32.4078341Z mkdir -p .additional_ci_files 2025-10-10T01:31:32.4078647Z mv td_results.json .additional_ci_files/td_results.json || true 2025-10-10T01:31:32.4087405Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:32.4087686Z env: 2025-10-10T01:31:32.4087847Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:32.4088109Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:32.4088390Z ##[endgroup] 2025-10-10T01:31:32.4151514Z mv: cannot stat 'td_results.json': No such file or directory 2025-10-10T01:31:32.4190314Z ##[group]Run .github/scripts/parse_ref.py 2025-10-10T01:31:32.4190616Z .github/scripts/parse_ref.py 2025-10-10T01:31:32.4198445Z shell: /usr/bin/bash -e {0} 2025-10-10T01:31:32.4198649Z env: 2025-10-10T01:31:32.4198813Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:32.4199150Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:32.4199426Z ##[endgroup] 2025-10-10T01:31:32.4423909Z Setting output branch=main 2025-10-10T01:31:32.4527400Z Prepare all required actions 2025-10-10T01:31:32.4527741Z Getting action download info 2025-10-10T01:31:32.6029287Z ##[group]Run ./.github/actions/filter-test-configs 2025-10-10T01:31:32.6029550Z with: 2025-10-10T01:31:32.6029933Z github-token: *** 2025-10-10T01:31:32.6034659Z test-matrix: {"include": [{"config": "vllm_basic_correctness_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_basic_models_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_entrypoints_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_regression_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_processor_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_pytorch_compilation_unit_tests", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_languagde_model_test_extended_generation_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_distributed_test_2_gpu_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 0, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 1, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 2, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 3, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_tp_test_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}, {"config": "vllm_distributed_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}]} 2025-10-10T01:31:32.6040068Z job-name: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:31:32.6040449Z env: 2025-10-10T01:31:32.6040608Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:32.6040850Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:32.6041096Z ##[endgroup] 2025-10-10T01:31:32.6079562Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T01:31:32.6079789Z with: 2025-10-10T01:31:32.6079939Z shell: bash 2025-10-10T01:31:32.6080101Z timeout_minutes: 10 2025-10-10T01:31:32.6080283Z max_attempts: 5 2025-10-10T01:31:32.6080453Z retry_wait_seconds: 30 2025-10-10T01:31:32.6081012Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-10-10T01:31:32.6081581Z polling_interval_seconds: 1 2025-10-10T01:31:32.6081782Z warning_on_retry: true 2025-10-10T01:31:32.6081969Z continue_on_error: false 2025-10-10T01:31:32.6082149Z env: 2025-10-10T01:31:32.6082289Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:32.6082514Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:32.6082895Z GITHUB_TOKEN: *** 2025-10-10T01:31:32.6083069Z ##[endgroup] 2025-10-10T01:31:32.7068200Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-10-10T01:31:32.9226439Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T01:31:33.0299319Z Collecting requests==2.27.1 2025-10-10T01:31:33.0500859Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-10-10T01:31:33.2051875Z Collecting pyyaml==6.0.2 2025-10-10T01:31:33.2090745Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-10-10T01:31:33.5445311Z Collecting charset-normalizer~=2.0.0 2025-10-10T01:31:33.5537871Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-10-10T01:31:33.6019504Z Collecting certifi>=2017.4.17 2025-10-10T01:31:33.6063955Z Downloading certifi-2025.10.5-py3-none-any.whl (163 kB) 2025-10-10T01:31:33.6135578Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-10-10T01:31:33.6143164Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-10-10T01:31:33.6882505Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-10-10T01:31:33.8007162Z Successfully installed certifi-2025.10.5 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-10-10T01:31:34.6846030Z Command completed after 1 attempt(s). 2025-10-10T01:31:34.6911197Z ##[group]Run set -x 2025-10-10T01:31:34.6911434Z set -x 2025-10-10T01:31:34.6911596Z  2025-10-10T01:31:34.6911866Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-10-10T01:31:34.6912206Z # in runner workspace 2025-10-10T01:31:34.6912546Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-10-10T01:31:34.6920598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:34.6921084Z env: 2025-10-10T01:31:34.6921256Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:34.6921510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:34.6921767Z ##[endgroup] 2025-10-10T01:31:34.6952787Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-10-10T01:31:34.7121194Z Setting output branch=main 2025-10-10T01:31:34.7182886Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-10-10T01:31:34.7183200Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-10-10T01:31:34.7183620Z echo "Job name: ${JOB_NAME}" 2025-10-10T01:31:34.7183829Z  2025-10-10T01:31:34.7184093Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-10-10T01:31:34.7184457Z # in runner workspace 2025-10-10T01:31:34.7184763Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-10-10T01:31:34.7185103Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-10-10T01:31:34.7185344Z  --job-name "${JOB_NAME}" \ 2025-10-10T01:31:34.7190273Z  --test-matrix "{"include": [{"config": "vllm_basic_correctness_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_basic_models_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_entrypoints_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_regression_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_processor_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_pytorch_compilation_unit_tests", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_languagde_model_test_extended_generation_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_distributed_test_2_gpu_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 0, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 1, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 2, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 3, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_tp_test_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}, {"config": "vllm_distributed_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}]}" \ 2025-10-10T01:31:34.7195119Z  --selected-test-configs "" \ 2025-10-10T01:31:34.7195353Z  --pr-number "${PR_NUMBER}" \ 2025-10-10T01:31:34.7195572Z  --tag "${TAG}" \ 2025-10-10T01:31:34.7195781Z  --event-name "${EVENT_NAME}" \ 2025-10-10T01:31:34.7196008Z  --schedule "${SCHEDULE}" \ 2025-10-10T01:31:34.7196487Z  --branch "${HEAD_BRANCH}" 2025-10-10T01:31:34.7203891Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:34.7204176Z env: 2025-10-10T01:31:34.7204346Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:34.7204593Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:34.7205199Z GITHUB_TOKEN: *** 2025-10-10T01:31:34.7205562Z JOB_NAME: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:31:34.7206138Z PR_NUMBER: 2025-10-10T01:31:34.7206311Z TAG: 2025-10-10T01:31:34.7206468Z EVENT_NAME: push 2025-10-10T01:31:34.7206632Z SCHEDULE: 2025-10-10T01:31:34.7206791Z HEAD_BRANCH: main 2025-10-10T01:31:34.7206959Z ##[endgroup] 2025-10-10T01:31:34.7235132Z Workflow: vllm-test 2025-10-10T01:31:34.7235512Z Job name: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:31:34.9020145Z Setting output keep-going=True 2025-10-10T01:31:34.9020690Z Setting output ci-verbose-test-logs=False 2025-10-10T01:31:34.9021176Z Setting output ci-test-showlocals=False 2025-10-10T01:31:34.9021478Z Setting output ci-no-test-timeout=False 2025-10-10T01:31:34.9021773Z Setting output ci-no-td=False 2025-10-10T01:31:34.9022042Z Setting output ci-td-distributed=False 2025-10-10T01:31:34.9022330Z Setting output is-unstable=False 2025-10-10T01:31:34.9022597Z Setting output reenabled-issues= 2025-10-10T01:31:34.9028479Z Setting output test-matrix={"include": [{"config": "vllm_basic_correctness_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_basic_models_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_entrypoints_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_regression_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_processor_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_pytorch_compilation_unit_tests", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_languagde_model_test_extended_generation_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_distributed_test_2_gpu_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 0, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 1, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 2, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 3, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_tp_test_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}, {"config": "vllm_distributed_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}]} 2025-10-10T01:31:34.9033333Z Setting output is-test-matrix-empty=False 2025-10-10T01:31:34.9138078Z ##[group]Run echo "Filtered matrix:" 2025-10-10T01:31:34.9138349Z echo "Filtered matrix:" 2025-10-10T01:31:34.9143009Z echo "{"include": [{"config": "vllm_basic_correctness_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_basic_models_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_entrypoints_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_regression_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_processor_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_pytorch_compilation_unit_tests", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_multi_model_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_languagde_model_test_extended_generation_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_distributed_test_2_gpu_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 0, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 1, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 2, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_test", "shard": 3, "num_shards": 4, "runner": "linux.g6.4xlarge.experimental.nvidia.gpu"}, {"config": "vllm_lora_tp_test_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}, {"config": "vllm_distributed_test_28_failure_test", "shard": 1, "num_shards": 1, "runner": "linux.g6.12xlarge.nvidia.gpu"}]}" 2025-10-10T01:31:34.9148004Z  2025-10-10T01:31:34.9148163Z echo 2025-10-10T01:31:34.9148368Z echo "Is the current job unstable? False" 2025-10-10T01:31:34.9148607Z  2025-10-10T01:31:34.9148746Z echo 2025-10-10T01:31:34.9148928Z echo "Is keep-going label set? True" 2025-10-10T01:31:34.9149161Z  2025-10-10T01:31:34.9149304Z echo 2025-10-10T01:31:34.9149473Z echo "Reenabled issues? " 2025-10-10T01:31:34.9160759Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:34.9161260Z env: 2025-10-10T01:31:34.9161549Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:34.9161898Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:34.9162175Z ##[endgroup] 2025-10-10T01:31:34.9194861Z Filtered matrix: 2025-10-10T01:31:34.9205096Z {include: [{config: vllm_basic_correctness_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_basic_models_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_entrypoints_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_regression_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_multi_model_processor_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_pytorch_compilation_unit_tests, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_lora_28_failure_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_multi_model_test_28_failure_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_languagde_model_test_extended_generation_28_failure_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_distributed_test_2_gpu_28_failure_test, shard: 1, num_shards: 1, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_lora_test, shard: 0, num_shards: 4, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_lora_test, shard: 1, num_shards: 4, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_lora_test, shard: 2, num_shards: 4, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_lora_test, shard: 3, num_shards: 4, runner: linux.g6.4xlarge.experimental.nvidia.gpu}, {config: vllm_lora_tp_test_distributed, shard: 1, num_shards: 1, runner: linux.g6.12xlarge.nvidia.gpu}, {config: vllm_distributed_test_28_failure_test, shard: 1, num_shards: 1, runner: linux.g6.12xlarge.nvidia.gpu}]} 2025-10-10T01:31:34.9213953Z 2025-10-10T01:31:34.9214147Z Is the current job unstable? False 2025-10-10T01:31:34.9214484Z 2025-10-10T01:31:34.9214628Z Is keep-going label set? True 2025-10-10T01:31:34.9214871Z 2025-10-10T01:31:34.9215221Z Reenabled issues? 2025-10-10T01:31:34.9268509Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-10-10T01:31:34.9268938Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-10-10T01:31:34.9275947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:34.9276229Z env: 2025-10-10T01:31:34.9276392Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:34.9276635Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:34.9276885Z JOB_TIMEOUT: 240 2025-10-10T01:31:34.9277052Z ##[endgroup] 2025-10-10T01:31:34.9343470Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T01:31:34.9343991Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T01:31:34.9344446Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T01:31:34.9351799Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:31:34.9352257Z env: 2025-10-10T01:31:34.9352532Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:34.9352860Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:34.9353240Z ##[endgroup] 2025-10-10T01:31:34.9471254Z ##[group]Run set -x 2025-10-10T01:31:34.9471805Z set -x 2025-10-10T01:31:34.9472051Z  2025-10-10T01:31:34.9472386Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-10-10T01:31:34.9472760Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-10-10T01:31:34.9473128Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-10-10T01:31:34.9473555Z  TEST_COMMAND=.ci/onnx/test.sh 2025-10-10T01:31:34.9473828Z else 2025-10-10T01:31:34.9474096Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-10-10T01:31:34.9474500Z fi 2025-10-10T01:31:34.9474740Z  2025-10-10T01:31:34.9474968Z # Leaving 1GB for the runner and other things 2025-10-10T01:31:34.9475550Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-10-10T01:31:34.9476334Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-10-10T01:31:34.9476984Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-10-10T01:31:34.9477445Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-10-10T01:31:34.9477809Z  2025-10-10T01:31:34.9478160Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-10-10T01:31:34.9478493Z  SHM_OPTS= 2025-10-10T01:31:34.9478818Z  JENKINS_USER= 2025-10-10T01:31:34.9479251Z  # ensure that docker container cleanly exits in 12 hours 2025-10-10T01:31:34.9479702Z  # if for some reason cleanup action doesn't stop container 2025-10-10T01:31:34.9480115Z  # when job is cancelled 2025-10-10T01:31:34.9480438Z  DOCKER_SHELL_CMD="sleep 12h" 2025-10-10T01:31:34.9480747Z else 2025-10-10T01:31:34.9481024Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-10-10T01:31:34.9481379Z  JENKINS_USER="--user jenkins" 2025-10-10T01:31:34.9481699Z  DOCKER_SHELL_CMD= 2025-10-10T01:31:34.9482019Z fi 2025-10-10T01:31:34.9482221Z  2025-10-10T01:31:34.9482559Z # detached container should get cleaned up by teardown_ec2_linux 2025-10-10T01:31:34.9483066Z # TODO: Stop building test binaries as part of the build phase 2025-10-10T01:31:34.9483603Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-10-10T01:31:34.9484043Z # shellcheck disable=SC2086,SC2090 2025-10-10T01:31:34.9484415Z container_name=$(docker run \ 2025-10-10T01:31:34.9484738Z  ${GPU_FLAG:-} \ 2025-10-10T01:31:34.9485026Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-10-10T01:31:34.9485450Z  -e BUILD_ENVIRONMENT \ 2025-10-10T01:31:34.9485714Z  -e PR_NUMBER \ 2025-10-10T01:31:34.9486146Z  -e GITHUB_ACTIONS \ 2025-10-10T01:31:34.9486533Z  -e GITHUB_REPOSITORY \ 2025-10-10T01:31:34.9486866Z  -e GITHUB_WORKFLOW \ 2025-10-10T01:31:34.9487190Z  -e GITHUB_JOB \ 2025-10-10T01:31:34.9487472Z  -e GITHUB_RUN_ID \ 2025-10-10T01:31:34.9487749Z  -e GITHUB_RUN_NUMBER \ 2025-10-10T01:31:34.9488088Z  -e GITHUB_RUN_ATTEMPT \ 2025-10-10T01:31:34.9488396Z  -e JOB_ID \ 2025-10-10T01:31:34.9488626Z  -e JOB_NAME \ 2025-10-10T01:31:34.9488854Z  -e BASE_SHA \ 2025-10-10T01:31:34.9489029Z  -e BRANCH \ 2025-10-10T01:31:34.9489202Z  -e SHA1 \ 2025-10-10T01:31:34.9489386Z  -e AWS_DEFAULT_REGION \ 2025-10-10T01:31:34.9489591Z  -e IN_WHEEL_TEST \ 2025-10-10T01:31:34.9489778Z  -e SHARD_NUMBER \ 2025-10-10T01:31:34.9489971Z  -e TEST_CONFIG \ 2025-10-10T01:31:34.9490166Z  -e NUM_TEST_SHARDS \ 2025-10-10T01:31:34.9490379Z  -e REENABLED_ISSUES \ 2025-10-10T01:31:34.9490590Z  -e CONTINUE_THROUGH_ERROR \ 2025-10-10T01:31:34.9491075Z  -e VERBOSE_TEST_LOGS \ 2025-10-10T01:31:34.9491335Z  -e TEST_SHOWLOCALS \ 2025-10-10T01:31:34.9491538Z  -e NO_TEST_TIMEOUT \ 2025-10-10T01:31:34.9491723Z  -e NO_TD \ 2025-10-10T01:31:34.9491908Z  -e TD_DISTRIBUTED \ 2025-10-10T01:31:34.9492215Z  -e PR_LABELS \ 2025-10-10T01:31:34.9492436Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-10-10T01:31:34.9492678Z  -e SCCACHE_BUCKET \ 2025-10-10T01:31:34.9492870Z  -e SCCACHE_REGION \ 2025-10-10T01:31:34.9493152Z  -e XLA_CUDA \ 2025-10-10T01:31:34.9493443Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-10-10T01:31:34.9493699Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-10-10T01:31:34.9493954Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-10-10T01:31:34.9494296Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-10-10T01:31:34.9494619Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-10-10T01:31:34.9494892Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-10-10T01:31:34.9495130Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-10-10T01:31:34.9495364Z  -e DASHBOARD_TAG \ 2025-10-10T01:31:34.9495578Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-10-10T01:31:34.9495841Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-10-10T01:31:34.9496433Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-10-10T01:31:34.9496749Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-10-10T01:31:34.9497045Z  --security-opt seccomp=unconfined \ 2025-10-10T01:31:34.9497292Z  --cap-add=SYS_PTRACE \ 2025-10-10T01:31:34.9497501Z  --ipc=host \ 2025-10-10T01:31:34.9497682Z  ${SHM_OPTS} \ 2025-10-10T01:31:34.9497880Z  --tty \ 2025-10-10T01:31:34.9498056Z  --detach \ 2025-10-10T01:31:34.9498255Z  --name="${container_name}" \ 2025-10-10T01:31:34.9498472Z  ${JENKINS_USER} \ 2025-10-10T01:31:34.9498723Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-10-10T01:31:34.9499004Z  -w /var/lib/jenkins/workspace \ 2025-10-10T01:31:34.9499231Z  "${DOCKER_IMAGE}" \ 2025-10-10T01:31:34.9499421Z  ${DOCKER_SHELL_CMD} 2025-10-10T01:31:34.9499604Z ) 2025-10-10T01:31:34.9499841Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-10-10T01:31:34.9500130Z  2025-10-10T01:31:34.9500311Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-10-10T01:31:34.9500723Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-10-10T01:31:34.9501095Z fi 2025-10-10T01:31:34.9501248Z  2025-10-10T01:31:34.9501591Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-10-10T01:31:34.9508795Z shell: /usr/bin/bash -e {0} 2025-10-10T01:31:34.9509000Z env: 2025-10-10T01:31:34.9509164Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:31:34.9509410Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:31:34.9509725Z BUILD_ENVIRONMENT: linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T01:31:34.9509982Z PR_NUMBER: 2025-10-10T01:31:34.9510161Z GITHUB_REPOSITORY: pytorch/pytorch 2025-10-10T01:31:34.9510381Z GITHUB_WORKFLOW: vllm-test 2025-10-10T01:31:34.9510564Z GITHUB_JOB: test 2025-10-10T01:31:34.9510730Z GITHUB_RUN_ID: 18392306035 2025-10-10T01:31:34.9510913Z GITHUB_RUN_NUMBER: 2144 2025-10-10T01:31:34.9511096Z GITHUB_RUN_ATTEMPT: 1 2025-10-10T01:31:34.9511263Z JOB_ID: 52408504898 2025-10-10T01:31:34.9511600Z JOB_NAME: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:31:34.9511965Z BRANCH: main 2025-10-10T01:31:34.9512151Z SHA1: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:31:34.9512420Z BASE_SHA: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:31:34.9512826Z TEST_CONFIG: vllm_basic_models_test 2025-10-10T01:31:34.9513053Z SHARD_NUMBER: 1 2025-10-10T01:31:34.9513218Z NUM_TEST_SHARDS: 1 2025-10-10T01:31:34.9513381Z EXTRA_FLAGS: 2025-10-10T01:31:34.9513538Z OP_BENCHMARK_TESTS: 2025-10-10T01:31:34.9513710Z REENABLED_ISSUES: 2025-10-10T01:31:34.9513883Z CONTINUE_THROUGH_ERROR: True 2025-10-10T01:31:34.9514080Z VERBOSE_TEST_LOGS: False 2025-10-10T01:31:34.9514261Z TEST_SHOWLOCALS: False 2025-10-10T01:31:34.9514444Z NO_TEST_TIMEOUT: False 2025-10-10T01:31:34.9514617Z NO_TD: False 2025-10-10T01:31:34.9514777Z TD_DISTRIBUTED: False 2025-10-10T01:31:34.9514994Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-10-10T01:31:34.9515250Z SCCACHE_REGION: us-east-1 2025-10-10T01:31:34.9515436Z SHM_SIZE: 2g 2025-10-10T01:31:34.9516028Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:31:34.9516639Z XLA_CUDA: 2025-10-10T01:31:34.9516916Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-10-10T01:31:34.9517244Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-10-10T01:31:34.9517477Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-10-10T01:31:34.9517682Z DASHBOARD_TAG: 2025-10-10T01:31:34.9518007Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-10-10T01:31:34.9518308Z HUGGING_FACE_HUB_TOKEN: *** 2025-10-10T01:31:34.9518619Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-10-10T01:31:34.9519118Z ARTIFACTS_FILE_SUFFIX: test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T01:31:34.9519533Z ##[endgroup] 2025-10-10T01:31:34.9545497Z + [[ vllm_basic_models_test == \m\u\l\t\i\g\p\u ]] 2025-10-10T01:31:34.9545798Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *onnx* ]] 2025-10-10T01:31:34.9546069Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-10-10T01:31:34.9549507Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-10-10T01:31:34.9573992Z + TOTAL_AVAILABLE_MEMORY_IN_GB='59.453 ' 2025-10-10T01:31:34.9574301Z + TOTAL_MEMORY_WITH_SWAP=62 2025-10-10T01:31:34.9574614Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *\s\3\9\0\x* ]] 2025-10-10T01:31:34.9574960Z + SHM_OPTS=--shm-size=2g 2025-10-10T01:31:34.9575205Z + JENKINS_USER='--user jenkins' 2025-10-10T01:31:34.9575448Z + DOCKER_SHELL_CMD= 2025-10-10T01:31:34.9583313Z +++ nproc --ignore=2 2025-10-10T01:31:34.9619728Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=59g --memory-swap=62g --env-file=/tmp/github_env_18392306035 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:31:56.0067804Z + container_name=3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T01:31:56.0068401Z + echo DOCKER_CONTAINER_ID=3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T01:31:56.0069147Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *\s\3\9\0\x* ]] 2025-10-10T01:31:56.0075533Z ++ echo dist/torch-2.10.0a0+git344e636-cp312-cp312-linux_x86_64.whl 2025-10-10T01:31:56.0077926Z + docker exec -t 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f sh -c 'python3 -m pip install dist/torch-2.10.0a0+git344e636-cp312-cp312-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-10-10T01:31:56.4776723Z Processing ./dist/torch-2.10.0a0+git344e636-cp312-cp312-linux_x86_64.whl (from torch==2.10.0a0+git344e636) 2025-10-10T01:31:56.8695653Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.18.0) 2025-10-10T01:31:56.8697550Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (4.12.2) 2025-10-10T01:31:56.8701903Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (78.1.1) 2025-10-10T01:31:56.8704873Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (1.13.3) 2025-10-10T01:31:56.8707814Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (2.8.8) 2025-10-10T01:31:56.8710844Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.1.6) 2025-10-10T01:31:56.8713870Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (2025.9.0) 2025-10-10T01:31:56.8725902Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.3.0) 2025-10-10T01:31:56.8823387Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (1.26.2) 2025-10-10T01:31:56.8861660Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (1.3.0) 2025-10-10T01:31:56.8905483Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2->torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.0.3) 2025-10-10T01:31:57.0883625Z Installing collected packages: torch 2025-10-10T01:32:09.0361283Z Successfully installed torch-2.10.0a0+git344e636 2025-10-10T01:32:09.0999325Z + export TERM=vt100 2025-10-10T01:32:09.1000025Z + TERM=vt100 2025-10-10T01:32:09.1001986Z ++ dirname .ci/pytorch/test.sh 2025-10-10T01:32:09.1012733Z + source .ci/pytorch/common.sh 2025-10-10T01:32:09.1016406Z +++ dirname .ci/pytorch/common.sh 2025-10-10T01:32:09.1025118Z ++ source .ci/pytorch/common_utils.sh 2025-10-10T01:32:09.1026796Z +++ declare -f -t trap_add 2025-10-10T01:32:09.1031407Z ++ set -ex -o pipefail 2025-10-10T01:32:09.1031648Z ++ [[ linux-jammy-cuda12.8-py3.12-gcc11 == *rocm* ]] 2025-10-10T01:32:09.1032018Z ++ BUILD_TEST_LIBTORCH=0 2025-10-10T01:32:09.1035329Z ++ dirname .ci/pytorch/test.sh 2025-10-10T01:32:09.1043381Z + source .ci/pytorch/common-build.sh 2025-10-10T01:32:09.1044976Z ++ [[ linux-jammy-cuda12.8-py3.12-gcc11 != *win-* ]] 2025-10-10T01:32:09.1051095Z ++++ dirname .ci/pytorch/common-build.sh 2025-10-10T01:32:09.1059545Z +++ cd .ci/pytorch 2025-10-10T01:32:09.1059777Z +++ pwd -P 2025-10-10T01:32:09.1062418Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-10-10T01:32:09.1062849Z ++ [[ linux-jammy-cuda12.8-py3.12-gcc11 == *-pch* ]] 2025-10-10T01:32:09.1063169Z ++ which sccache 2025-10-10T01:32:09.1110531Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-10-10T01:32:09.1111139Z ++ sccache --stop-server 2025-10-10T01:32:09.1140557Z ++ true 2025-10-10T01:32:09.1140853Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-10-10T01:32:09.1152559Z ++ trap_add sccache_epilogue EXIT 2025-10-10T01:32:09.1152851Z ++ trap_add_cmd=sccache_epilogue 2025-10-10T01:32:09.1153096Z ++ shift 2025-10-10T01:32:09.1153297Z ++ for trap_add_name in "$@" 2025-10-10T01:32:09.1159050Z ++++ trap -p EXIT 2025-10-10T01:32:09.1162417Z +++ eval 'extract_trap_cmd ' 2025-10-10T01:32:09.1162872Z ++++ extract_trap_cmd 2025-10-10T01:32:09.1163248Z ++++ printf '%s\n' '' 2025-10-10T01:32:09.1163634Z +++ printf '%s\n' sccache_epilogue 2025-10-10T01:32:09.1165608Z ++ trap -- ' 2025-10-10T01:32:09.1165905Z sccache_epilogue' EXIT 2025-10-10T01:32:09.1166210Z ++ [[ -n 1 ]] 2025-10-10T01:32:09.1166679Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-10-10T01:32:09.1167249Z Skipping sccache server initialization, setting environment variables 2025-10-10T01:32:09.1167648Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-10-10T01:32:09.1167911Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-10-10T01:32:09.1168214Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T01:32:09.1168594Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T01:32:09.1174970Z ++ export RUST_LOG=sccache::server=error 2025-10-10T01:32:09.1175254Z ++ RUST_LOG=sccache::server=error 2025-10-10T01:32:09.1175483Z ++ sccache --zero-stats 2025-10-10T01:32:09.5099716Z Statistics zeroed. 2025-10-10T01:32:09.5110652Z ++ which ccache 2025-10-10T01:32:09.5143161Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 != *rocm* ]] 2025-10-10T01:32:09.5143474Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 != *s390x* ]] 2025-10-10T01:32:09.5143868Z + [[ -d /var/lib/jenkins/workspace ]] 2025-10-10T01:32:09.5147077Z ++ stat -c %u /var/lib/jenkins/workspace 2025-10-10T01:32:09.5163769Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-10-10T01:32:09.5164090Z + trap_add cleanup_workspace EXIT 2025-10-10T01:32:09.5164349Z + trap_add_cmd=cleanup_workspace 2025-10-10T01:32:09.5164563Z + shift 2025-10-10T01:32:09.5164726Z + for trap_add_name in "$@" 2025-10-10T01:32:09.5171197Z +++ trap -p EXIT 2025-10-10T01:32:09.5174499Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-10-10T01:32:09.5174989Z sccache_epilogue'\'' EXIT' 2025-10-10T01:32:09.5175288Z +++ extract_trap_cmd trap -- ' 2025-10-10T01:32:09.5175624Z sccache_epilogue' EXIT 2025-10-10T01:32:09.5175886Z +++ printf '%s\n' ' 2025-10-10T01:32:09.5176105Z sccache_epilogue' 2025-10-10T01:32:09.5176354Z ++ printf '%s\n' cleanup_workspace 2025-10-10T01:32:09.5177328Z + trap -- ' 2025-10-10T01:32:09.5177546Z sccache_epilogue 2025-10-10T01:32:09.5177811Z cleanup_workspace' EXIT 2025-10-10T01:32:09.5178183Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-10-10T01:32:10.4993432Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-10-10T01:32:10.5018474Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *cuda* ]] 2025-10-10T01:32:10.5021964Z ++ python -c 'import os;import numba.cuda; print(os.path.dirname(numba.cuda.__file__))' 2025-10-10T01:32:10.9467153Z + NUMBA_CUDA_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/numba/cuda 2025-10-10T01:32:10.9467777Z + '[' -n /opt/conda/envs/py_3.12/lib/python3.12/site-packages/numba/cuda ']' 2025-10-10T01:32:10.9473139Z +++ realpath .ci/pytorch/test.sh 2025-10-10T01:32:10.9486032Z ++ dirname /var/lib/jenkins/workspace/.ci/pytorch/test.sh 2025-10-10T01:32:10.9644677Z + NUMBA_PATCH=/var/lib/jenkins/workspace/.ci/pytorch/numba-cuda-13.patch 2025-10-10T01:32:10.9645288Z + pushd /opt/conda/envs/py_3.12/lib/python3.12/site-packages/numba/cuda 2025-10-10T01:32:10.9645875Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/numba/cuda ~/workspace 2025-10-10T01:32:10.9646419Z + patch -p4 2025-10-10T01:32:10.9660614Z patching file cudadrv/driver.py 2025-10-10T01:32:10.9664545Z + popd 2025-10-10T01:32:10.9664760Z ~/workspace 2025-10-10T01:32:10.9665194Z + echo 'Environment variables:' 2025-10-10T01:32:10.9665605Z Environment variables: 2025-10-10T01:32:10.9665836Z + env 2025-10-10T01:32:10.9675733Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T01:32:10.9676339Z CONTINUE_THROUGH_ERROR=True 2025-10-10T01:32:10.9676670Z BUILD_ENVIRONMENT=linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T01:32:10.9677400Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-10-10T01:32:10.9677817Z HOSTNAME=3e581cf95cad 2025-10-10T01:32:10.9678373Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:10.9678817Z GITHUB_ACTION=__run_2 2025-10-10T01:32:10.9679010Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-10-10T01:32:10.9679337Z GITHUB_RUN_NUMBER=2144 2025-10-10T01:32:10.9679518Z TEST_CONFIG=vllm_basic_models_test 2025-10-10T01:32:10.9679734Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-10-10T01:32:10.9679968Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-10-10T01:32:10.9680200Z SCCACHE_IDLE_TIMEOUT=0 2025-10-10T01:32:10.9680502Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-10-10T01:32:10.9680746Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-10-10T01:32:10.9680977Z GITHUB_REF_TYPE=branch 2025-10-10T01:32:10.9681190Z BASE_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:10.9681424Z XLA_CUDA= 2025-10-10T01:32:10.9681591Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-10-10T01:32:10.9681886Z HUGGING_FACE_HUB_TOKEN=*** 2025-10-10T01:32:10.9682283Z *** 2025-10-10T01:32:10.9682445Z GITHUB_REPOSITORY_ID=65600975 2025-10-10T01:32:10.9682694Z GITHUB_ACTIONS=true 2025-10-10T01:32:10.9682877Z NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:32:10.9683123Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T01:32:10.9683517Z SHA1=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:10.9683893Z GITHUB_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:10.9684253Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/vllm.yml@refs/heads/main 2025-10-10T01:32:10.9684590Z UCC_HOME=/usr 2025-10-10T01:32:10.9684757Z VERBOSE_TEST_LOGS=False 2025-10-10T01:32:10.9684944Z GITHUB_REF=refs/heads/main 2025-10-10T01:32:10.9685131Z SHARD_NUMBER=1 2025-10-10T01:32:10.9685312Z GITHUB_REF_PROTECTED=true 2025-10-10T01:32:10.9685499Z HOME=/var/lib/jenkins 2025-10-10T01:32:10.9685703Z GITHUB_API_URL=https://api.github.com 2025-10-10T01:32:10.9685938Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-10-10T01:32:10.9686184Z UCX_COMMIT=7836b165abdbe468a2f607e7254011c07d788152 2025-10-10T01:32:10.9686418Z USE_SYSTEM_NCCL=1 2025-10-10T01:32:10.9686578Z NUM_TEST_SHARDS=1 2025-10-10T01:32:10.9686726Z UCX_HOME=/usr 2025-10-10T01:32:10.9687132Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:10.9687759Z JOB_NAME=ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:32:10.9688355Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:10.9689155Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-10-10T01:32:10.9689505Z GITHUB_EVENT_NAME=push 2025-10-10T01:32:10.9689682Z DASHBOARD_TAG= 2025-10-10T01:32:10.9689851Z GITHUB_RUN_ID=18392306035 2025-10-10T01:32:10.9690031Z INSTALLED_OPENBLAS= 2025-10-10T01:32:10.9690454Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:10.9690928Z GITHUB_ACTOR=pytorchmergebot 2025-10-10T01:32:10.9691119Z PR_NUMBER= 2025-10-10T01:32:10.9691271Z DESIRED_CUDA=12.8.1 2025-10-10T01:32:10.9691443Z GITHUB_RUN_ATTEMPT=1 2025-10-10T01:32:10.9691623Z ANACONDA_PYTHON_VERSION=3.12 2025-10-10T01:32:10.9691860Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-10-10T01:32:10.9692108Z TERM=vt100 2025-10-10T01:32:10.9692253Z INSTALLED_VISION=yes 2025-10-10T01:32:10.9692417Z BRANCH=main 2025-10-10T01:32:10.9692579Z SCCACHE_REGION=us-east-1 2025-10-10T01:32:10.9692775Z OPENSSL_ROOT_DIR=/opt/openssl 2025-10-10T01:32:10.9692966Z CUDA_PATH=/usr/local/cuda 2025-10-10T01:32:10.9693429Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-10-10T01:32:10.9693837Z GITHUB_SERVER_URL=https://github.com 2025-10-10T01:32:10.9694089Z UCC_COMMIT=430e241bf5d38cbc73fc7a6b89155397232e3f96 2025-10-10T01:32:10.9694321Z REENABLED_ISSUES= 2025-10-10T01:32:10.9694477Z DOCS= 2025-10-10T01:32:10.9694618Z SHLVL=1 2025-10-10T01:32:10.9694763Z MAX_JOBS=14 2025-10-10T01:32:10.9694910Z GITHUB_ACTOR_ID=97764156 2025-10-10T01:32:10.9695151Z GITHUB_WORKFLOW_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:10.9695436Z GITHUB_REF_NAME=main 2025-10-10T01:32:10.9695703Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-10-10T01:32:10.9695995Z GITHUB_JOB=test 2025-10-10T01:32:10.9696492Z NO_TEST_TIMEOUT=False 2025-10-10T01:32:10.9696789Z TD_DISTRIBUTED=False 2025-10-10T01:32:10.9696995Z GITHUB_REPOSITORY=pytorch/pytorch 2025-10-10T01:32:10.9697206Z GITHUB_RETENTION_DAYS=90 2025-10-10T01:32:10.9697426Z OPENSSL_DIR=/opt/openssl 2025-10-10T01:32:10.9697626Z GITHUB_ACTION_REPOSITORY= 2025-10-10T01:32:10.9698175Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T01:32:10.9698721Z GITHUB_BASE_REF= 2025-10-10T01:32:10.9698887Z INSTALLED_ACL= 2025-10-10T01:32:10.9699253Z ARTIFACTS_FILE_SUFFIX=test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T01:32:10.9699659Z CI=true 2025-10-10T01:32:10.9699826Z GITHUB_REPOSITORY_OWNER=pytorch 2025-10-10T01:32:10.9700074Z RUST_LOG=sccache::server=error 2025-10-10T01:32:10.9700281Z JOB_ID=52408504898 2025-10-10T01:32:10.9700443Z GITHUB_HEAD_REF= 2025-10-10T01:32:10.9700599Z GITHUB_ACTION_REF= 2025-10-10T01:32:10.9700802Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-10-10T01:32:10.9701057Z TEST_SHOWLOCALS=False 2025-10-10T01:32:10.9701237Z GITHUB_WORKFLOW=vllm-test 2025-10-10T01:32:10.9701429Z DEBIAN_FRONTEND=noninteractive 2025-10-10T01:32:10.9701866Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:10.9702304Z NO_TD=False 2025-10-10T01:32:10.9702467Z SKIP_SCCACHE_INITIALIZATION=1 2025-10-10T01:32:10.9702676Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-10-10T01:32:10.9702893Z _=/usr/bin/env 2025-10-10T01:32:10.9703143Z OLDPWD=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/numba/cuda 2025-10-10T01:32:10.9703514Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-10-10T01:32:10.9804139Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch 2025-10-10T01:32:10.9804944Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/bin 2025-10-10T01:32:10.9805455Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/lib 2025-10-10T01:32:10.9806199Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/test 2025-10-10T01:32:10.9806591Z + BUILD_DIR=build 2025-10-10T01:32:10.9806815Z + BUILD_RENAMED_DIR=build_renamed 2025-10-10T01:32:10.9807068Z + BUILD_BIN_DIR=build/bin 2025-10-10T01:32:10.9807295Z + SHARD_NUMBER=1 2025-10-10T01:32:10.9807655Z + NUM_TEST_SHARDS=1 2025-10-10T01:32:10.9820725Z + export TORCH_SERIALIZATION_DEBUG=1 2025-10-10T01:32:10.9821062Z + TORCH_SERIALIZATION_DEBUG=1 2025-10-10T01:32:10.9821281Z + export VALGRIND=ON 2025-10-10T01:32:10.9821469Z + VALGRIND=ON 2025-10-10T01:32:10.9821710Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *clang9* ]] 2025-10-10T01:32:10.9822002Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *xpu* ]] 2025-10-10T01:32:10.9822250Z + detect_cuda_arch 2025-10-10T01:32:10.9822459Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *cuda* ]] 2025-10-10T01:32:10.9822714Z + command -v nvidia-smi 2025-10-10T01:32:10.9823007Z /usr/bin/nvidia-smi 2025-10-10T01:32:10.9823377Z ++ nvidia-smi --query-gpu=compute_cap --format=csv 2025-10-10T01:32:10.9823742Z ++ tail -n 1 2025-10-10T01:32:11.0041444Z + TORCH_CUDA_ARCH_LIST=8.9 2025-10-10T01:32:11.0042088Z + export TORCH_CUDA_ARCH_LIST 2025-10-10T01:32:11.0042444Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *s390x* ]] 2025-10-10T01:32:11.0042756Z + [[ 0 == \1 ]] 2025-10-10T01:32:11.0042972Z + [[ True == \1 ]] 2025-10-10T01:32:11.0043220Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 != *bazel* ]] 2025-10-10T01:32:11.0045768Z ++ realpath build/custom_test_artifacts 2025-10-10T01:32:11.0185536Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-10-10T01:32:11.0185934Z + [[ -n '' ]] 2025-10-10T01:32:11.0186117Z + echo 'Environment variables' 2025-10-10T01:32:11.0186318Z Environment variables 2025-10-10T01:32:11.0186490Z + env 2025-10-10T01:32:11.0377479Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T01:32:11.0378282Z CONTINUE_THROUGH_ERROR=True 2025-10-10T01:32:11.0378885Z BUILD_ENVIRONMENT=linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T01:32:11.0379542Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-10-10T01:32:11.0379830Z HOSTNAME=3e581cf95cad 2025-10-10T01:32:11.0380420Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:11.0381108Z GITHUB_ACTION=__run_2 2025-10-10T01:32:11.0381392Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-10-10T01:32:11.0381646Z GITHUB_RUN_NUMBER=2144 2025-10-10T01:32:11.0381837Z TEST_CONFIG=vllm_basic_models_test 2025-10-10T01:32:11.0382054Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-10-10T01:32:11.0382410Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-10-10T01:32:11.0382638Z SCCACHE_IDLE_TIMEOUT=0 2025-10-10T01:32:11.0382933Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-10-10T01:32:11.0383285Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-10-10T01:32:11.0383521Z GITHUB_REF_TYPE=branch 2025-10-10T01:32:11.0383702Z TORCH_CUDA_ARCH_LIST=8.9 2025-10-10T01:32:11.0383954Z BASE_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:11.0384292Z XLA_CUDA= 2025-10-10T01:32:11.0384457Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-10-10T01:32:11.0384904Z HUGGING_FACE_HUB_TOKEN=*** 2025-10-10T01:32:11.0385221Z *** 2025-10-10T01:32:11.0385383Z GITHUB_REPOSITORY_ID=65600975 2025-10-10T01:32:11.0385601Z GITHUB_ACTIONS=true 2025-10-10T01:32:11.0385888Z NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T01:32:11.0386160Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T01:32:11.0386441Z SHA1=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:11.0386821Z GITHUB_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:11.0387191Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/vllm.yml@refs/heads/main 2025-10-10T01:32:11.0387565Z UCC_HOME=/usr 2025-10-10T01:32:11.0387812Z TORCH_SERIALIZATION_DEBUG=1 2025-10-10T01:32:11.0388104Z VERBOSE_TEST_LOGS=False 2025-10-10T01:32:11.0388301Z GITHUB_REF=refs/heads/main 2025-10-10T01:32:11.0388596Z SHARD_NUMBER=1 2025-10-10T01:32:11.0389011Z GITHUB_REF_PROTECTED=true 2025-10-10T01:32:11.0389198Z HOME=/var/lib/jenkins 2025-10-10T01:32:11.0389502Z GITHUB_API_URL=https://api.github.com 2025-10-10T01:32:11.0389780Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-10-10T01:32:11.0390029Z UCX_COMMIT=7836b165abdbe468a2f607e7254011c07d788152 2025-10-10T01:32:11.0390337Z USE_SYSTEM_NCCL=1 2025-10-10T01:32:11.0390557Z NUM_TEST_SHARDS=1 2025-10-10T01:32:11.0390717Z UCX_HOME=/usr 2025-10-10T01:32:11.0391162Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:11.0391859Z JOB_NAME=ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T01:32:11.0392582Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:11.0393258Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-10-10T01:32:11.0393603Z GITHUB_EVENT_NAME=push 2025-10-10T01:32:11.0393861Z DASHBOARD_TAG= 2025-10-10T01:32:11.0394081Z GITHUB_RUN_ID=18392306035 2025-10-10T01:32:11.0394268Z INSTALLED_OPENBLAS= 2025-10-10T01:32:11.0394941Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:11.0395505Z GITHUB_ACTOR=pytorchmergebot 2025-10-10T01:32:11.0395823Z PR_NUMBER= 2025-10-10T01:32:11.0395978Z DESIRED_CUDA=12.8.1 2025-10-10T01:32:11.0396356Z GITHUB_RUN_ATTEMPT=1 2025-10-10T01:32:11.0396660Z VALGRIND=ON 2025-10-10T01:32:11.0396832Z ANACONDA_PYTHON_VERSION=3.12 2025-10-10T01:32:11.0397080Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-10-10T01:32:11.0397368Z TERM=vt100 2025-10-10T01:32:11.0397598Z INSTALLED_VISION=yes 2025-10-10T01:32:11.0397770Z BRANCH=main 2025-10-10T01:32:11.0397930Z SCCACHE_REGION=us-east-1 2025-10-10T01:32:11.0398123Z OPENSSL_ROOT_DIR=/opt/openssl 2025-10-10T01:32:11.0398443Z CUDA_PATH=/usr/local/cuda 2025-10-10T01:32:11.0398824Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-10-10T01:32:11.0399588Z GITHUB_SERVER_URL=https://github.com 2025-10-10T01:32:11.0399853Z UCC_COMMIT=430e241bf5d38cbc73fc7a6b89155397232e3f96 2025-10-10T01:32:11.0400100Z REENABLED_ISSUES= 2025-10-10T01:32:11.0400265Z DOCS= 2025-10-10T01:32:11.0400409Z SHLVL=1 2025-10-10T01:32:11.0400555Z MAX_JOBS=14 2025-10-10T01:32:11.0400715Z GITHUB_ACTOR_ID=97764156 2025-10-10T01:32:11.0400963Z GITHUB_WORKFLOW_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:32:11.0401237Z GITHUB_REF_NAME=main 2025-10-10T01:32:11.0401501Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-10-10T01:32:11.0401801Z GITHUB_JOB=test 2025-10-10T01:32:11.0401972Z NO_TEST_TIMEOUT=False 2025-10-10T01:32:11.0402147Z TD_DISTRIBUTED=False 2025-10-10T01:32:11.0402330Z GITHUB_REPOSITORY=pytorch/pytorch 2025-10-10T01:32:11.0402543Z GITHUB_RETENTION_DAYS=90 2025-10-10T01:32:11.0402730Z OPENSSL_DIR=/opt/openssl 2025-10-10T01:32:11.0402919Z GITHUB_ACTION_REPOSITORY= 2025-10-10T01:32:11.0403468Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T01:32:11.0404019Z GITHUB_BASE_REF= 2025-10-10T01:32:11.0404214Z INSTALLED_ACL= 2025-10-10T01:32:11.0404581Z ARTIFACTS_FILE_SUFFIX=test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T01:32:11.0404997Z CI=true 2025-10-10T01:32:11.0405154Z GITHUB_REPOSITORY_OWNER=pytorch 2025-10-10T01:32:11.0405396Z RUST_LOG=sccache::server=error 2025-10-10T01:32:11.0405580Z JOB_ID=52408504898 2025-10-10T01:32:11.0405740Z GITHUB_HEAD_REF= 2025-10-10T01:32:11.0405914Z GITHUB_ACTION_REF= 2025-10-10T01:32:11.0406122Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-10-10T01:32:11.0406368Z TEST_SHOWLOCALS=False 2025-10-10T01:32:11.0406550Z GITHUB_WORKFLOW=vllm-test 2025-10-10T01:32:11.0406746Z DEBIAN_FRONTEND=noninteractive 2025-10-10T01:32:11.0407355Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_2c75cc07-934b-4496-b39b-4d489b429b58 2025-10-10T01:32:11.0407794Z NO_TD=False 2025-10-10T01:32:11.0407958Z SKIP_SCCACHE_INITIALIZATION=1 2025-10-10T01:32:11.0408176Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-10-10T01:32:11.0408495Z OLDPWD=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/numba/cuda 2025-10-10T01:32:11.0408776Z _=/usr/bin/env 2025-10-10T01:32:11.0408950Z + echo 'Testing pytorch' 2025-10-10T01:32:11.0409127Z Testing pytorch 2025-10-10T01:32:11.0409294Z + export LANG=C.UTF-8 2025-10-10T01:32:11.0409455Z + LANG=C.UTF-8 2025-10-10T01:32:11.0409608Z + PR_NUMBER= 2025-10-10T01:32:11.0409785Z + [[ vllm_basic_models_test == \d\e\f\a\u\l\t ]] 2025-10-10T01:32:11.0410060Z + [[ vllm_basic_models_test == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-10-10T01:32:11.0410313Z + [[ vllm_basic_models_test == \s\l\o\w ]] 2025-10-10T01:32:11.0410590Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *slow-gradcheck* ]] 2025-10-10T01:32:11.0410895Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *cuda* ]] 2025-10-10T01:32:11.0411160Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-10-10T01:32:11.0411521Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-10-10T01:32:11.0411754Z + [[ vllm_basic_models_test == *crossref* ]] 2025-10-10T01:32:11.0412003Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *rocm* ]] 2025-10-10T01:32:11.0412274Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *xpu* ]] 2025-10-10T01:32:11.0412549Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 != *-bazel-* ]] 2025-10-10T01:32:11.0412791Z + pip_install ninja==1.10.2 2025-10-10T01:32:11.0413063Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-10-10T01:32:11.0413385Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-10-10T01:32:11.4702899Z Collecting ninja==1.10.2 2025-10-10T01:32:11.4886794Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-10-10T01:32:11.5188417Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-10-10T01:32:11.7182234Z Installing collected packages: ninja 2025-10-10T01:32:11.7182630Z Attempting uninstall: ninja 2025-10-10T01:32:11.7210311Z Found existing installation: ninja 1.11.1.4 2025-10-10T01:32:11.7232871Z Uninstalling ninja-1.11.1.4: 2025-10-10T01:32:11.7360590Z Successfully uninstalled ninja-1.11.1.4 2025-10-10T01:32:11.7876710Z Successfully installed ninja-1.10.2 2025-10-10T01:32:11.8306958Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T01:32:11.8308538Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T01:32:11.8309543Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *aarch64* ]] 2025-10-10T01:32:11.8309988Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *asan* ]] 2025-10-10T01:32:11.8310342Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *-debug* ]] 2025-10-10T01:32:11.8310865Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 != *-bazel-* ]] 2025-10-10T01:32:11.8311400Z + echo 'We are not in debug mode: linux-jammy-cuda12.8-py3.12-gcc11. Expect the assertion to pass' 2025-10-10T01:32:11.8312065Z We are not in debug mode: linux-jammy-cuda12.8-py3.12-gcc11. Expect the assertion to pass 2025-10-10T01:32:11.8312492Z + cd test 2025-10-10T01:32:11.8312788Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-10-10T01:32:13.2922424Z + [[ vllm_basic_models_test == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-10-10T01:32:13.2922871Z + [[ vllm_basic_models_test == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-10-10T01:32:13.2923278Z + [[ vllm_basic_models_test == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-10-10T01:32:13.2926435Z + DYNAMO_BENCHMARK_FLAGS=() 2025-10-10T01:32:13.2927009Z + [[ vllm_basic_models_test == *pr_time_benchmarks* ]] 2025-10-10T01:32:13.2927723Z + [[ vllm_basic_models_test == *dynamo_eager* ]] 2025-10-10T01:32:13.2928042Z + [[ vllm_basic_models_test == *aot_eager* ]] 2025-10-10T01:32:13.2928342Z + [[ vllm_basic_models_test == *aot_inductor* ]] 2025-10-10T01:32:13.2928672Z + [[ vllm_basic_models_test == *max_autotune_inductor* ]] 2025-10-10T01:32:13.2929000Z + [[ vllm_basic_models_test == *inductor* ]] 2025-10-10T01:32:13.2929293Z + [[ vllm_basic_models_test == *dynamic* ]] 2025-10-10T01:32:13.2929582Z + [[ vllm_basic_models_test == *cpu* ]] 2025-10-10T01:32:13.2929875Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-10-10T01:32:13.3056878Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *libtorch* ]] 2025-10-10T01:32:13.3057283Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *-bazel-* ]] 2025-10-10T01:32:13.3060332Z + cd test 2025-10-10T01:32:13.3061252Z + python -c 'import torch; print(torch.__config__.show())' 2025-10-10T01:32:14.8485516Z PyTorch built with: 2025-10-10T01:32:14.8485806Z - GCC 11.4 2025-10-10T01:32:14.8486051Z - C++ Version: 201703 2025-10-10T01:32:14.8486891Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-10-10T01:32:14.8487582Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-10-10T01:32:14.8487969Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-10-10T01:32:14.8488276Z - LAPACK is enabled (usually provided by MKL) 2025-10-10T01:32:14.8488564Z - NNPACK is enabled 2025-10-10T01:32:14.8488796Z - CPU capability usage: AVX2 2025-10-10T01:32:14.8489033Z - CUDA Runtime 12.8 2025-10-10T01:32:14.8489574Z - NVCC architecture flags: -gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90 2025-10-10T01:32:14.8490127Z - CuDNN 90.8 2025-10-10T01:32:14.8494160Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e, CUDA_VERSION=12.8, CUDNN_VERSION=9.8.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-10-10T01:32:14.8497796Z 2025-10-10T01:32:15.1147246Z + cd test 2025-10-10T01:32:15.1147670Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-10-10T01:32:16.3406071Z ATen/Parallel: 2025-10-10T01:32:16.3406435Z at::get_num_threads() : 8 2025-10-10T01:32:16.3406714Z at::get_num_interop_threads() : 8 2025-10-10T01:32:16.3406992Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-10-10T01:32:16.3407254Z omp_get_max_threads() : 8 2025-10-10T01:32:16.3407767Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-10-10T01:32:16.3408287Z mkl_get_max_threads() : 8 2025-10-10T01:32:16.3408623Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-10-10T01:32:16.3409018Z std::thread::hardware_concurrency() : 16 2025-10-10T01:32:16.3409313Z Environment variables: 2025-10-10T01:32:16.3409543Z OMP_NUM_THREADS : [not set] 2025-10-10T01:32:16.3409778Z MKL_NUM_THREADS : [not set] 2025-10-10T01:32:16.3410029Z ATen parallel backend: OpenMP 2025-10-10T01:32:16.3410588Z 2025-10-10T01:32:16.5657635Z + [[ vllm_basic_models_test == *numpy_2* ]] 2025-10-10T01:32:16.5658097Z + [[ linux-jammy-cuda12.8-py3.12-gcc11 == *aarch64* ]] 2025-10-10T01:32:16.5658452Z + [[ vllm_basic_models_test == *backward* ]] 2025-10-10T01:32:16.5658738Z + [[ vllm_basic_models_test == *xla* ]] 2025-10-10T01:32:16.5659014Z + [[ vllm_basic_models_test == *vllm* ]] 2025-10-10T01:32:16.5659316Z + echo 'vLLM CI uses TORCH_CUDA_ARCH_LIST: 8.9' 2025-10-10T01:32:16.5659625Z vLLM CI uses TORCH_CUDA_ARCH_LIST: 8.9 2025-10-10T01:32:16.5660798Z + cd .ci/lumen_cli 2025-10-10T01:32:16.5661448Z + python -m pip install -e . 2025-10-10T01:32:16.8523929Z Obtaining file:///var/lib/jenkins/workspace/.ci/lumen_cli 2025-10-10T01:32:17.5774730Z Installing build dependencies ... [?25l- \ done 2025-10-10T01:32:17.7935808Z [?25h Checking if build backend supports build_editable ... [?25ldone 2025-10-10T01:32:18.0534927Z [?25h Getting requirements to build editable ... [?25l- done 2025-10-10T01:32:18.2400877Z [?25h Preparing editable metadata (pyproject.toml) ... [?25l- done 2025-10-10T01:32:18.2465779Z [?25hRequirement already satisfied: pyyaml==6.0.2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from lumen-ci==0.1.0) (6.0.2) 2025-10-10T01:32:18.2926306Z Collecting GitPython==3.1.45 (from lumen-ci==0.1.0) 2025-10-10T01:32:18.3129513Z Downloading gitpython-3.1.45-py3-none-any.whl.metadata (13 kB) 2025-10-10T01:32:18.3349063Z Collecting docker==7.1.0 (from lumen-ci==0.1.0) 2025-10-10T01:32:18.3381425Z Downloading docker-7.1.0-py3-none-any.whl.metadata (3.8 kB) 2025-10-10T01:32:18.3441892Z Requirement already satisfied: pytest==7.3.2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from lumen-ci==0.1.0) (7.3.2) 2025-10-10T01:32:18.5229651Z Collecting uv==0.8.6 (from lumen-ci==0.1.0) 2025-10-10T01:32:18.5324181Z Downloading uv-0.8.6-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB) 2025-10-10T01:32:18.5381141Z Requirement already satisfied: requests>=2.26.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from docker==7.1.0->lumen-ci==0.1.0) (2.32.5) 2025-10-10T01:32:18.5384192Z Requirement already satisfied: urllib3>=1.26.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from docker==7.1.0->lumen-ci==0.1.0) (2.5.0) 2025-10-10T01:32:18.5498306Z Collecting gitdb<5,>=4.0.1 (from GitPython==3.1.45->lumen-ci==0.1.0) 2025-10-10T01:32:18.5543305Z Downloading gitdb-4.0.12-py3-none-any.whl.metadata (1.2 kB) 2025-10-10T01:32:18.5606961Z Requirement already satisfied: iniconfig in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from pytest==7.3.2->lumen-ci==0.1.0) (2.1.0) 2025-10-10T01:32:18.5610304Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from pytest==7.3.2->lumen-ci==0.1.0) (23.1) 2025-10-10T01:32:18.5615031Z Requirement already satisfied: pluggy<2.0,>=0.12 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from pytest==7.3.2->lumen-ci==0.1.0) (1.6.0) 2025-10-10T01:32:18.5765244Z Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython==3.1.45->lumen-ci==0.1.0) 2025-10-10T01:32:18.5891315Z Downloading smmap-5.0.2-py3-none-any.whl.metadata (4.3 kB) 2025-10-10T01:32:18.5957667Z Requirement already satisfied: charset_normalizer<4,>=2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests>=2.26.0->docker==7.1.0->lumen-ci==0.1.0) (3.4.3) 2025-10-10T01:32:18.5961741Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests>=2.26.0->docker==7.1.0->lumen-ci==0.1.0) (3.10) 2025-10-10T01:32:18.5967062Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests>=2.26.0->docker==7.1.0->lumen-ci==0.1.0) (2025.10.5) 2025-10-10T01:32:18.6057544Z Downloading docker-7.1.0-py3-none-any.whl (147 kB) 2025-10-10T01:32:18.6174634Z Downloading gitpython-3.1.45-py3-none-any.whl (208 kB) 2025-10-10T01:32:18.6322243Z Downloading uv-0.8.6-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB) 2025-10-10T01:32:18.8353026Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/19.3 MB ? eta -:--:-- 2025-10-10T01:32:18.8818472Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 18.1/19.3 MB 89.4 MB/s eta 0:00:01 2025-10-10T01:32:18.8819218Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 77.1 MB/s 0:00:00 2025-10-10T01:32:18.8903866Z [?25hDownloading gitdb-4.0.12-py3-none-any.whl (62 kB) 2025-10-10T01:32:18.8981153Z Downloading smmap-5.0.2-py3-none-any.whl (24 kB) 2025-10-10T01:32:18.9199016Z Building wheels for collected packages: lumen-ci 2025-10-10T01:32:19.1014012Z Building editable for lumen-ci (pyproject.toml) ... [?25l- done 2025-10-10T01:32:19.1020842Z [?25h Created wheel for lumen-ci: filename=lumen_ci-0.1.0-0.editable-py3-none-any.whl size=2709 sha256=eca049b570da8dbc4bb828be51e6c7b467b62c85485f7d3dd18ec6b0de1b9069 2025-10-10T01:32:19.1022437Z Stored in directory: /tmp/pip-ephem-wheel-cache-zwxrl_38/wheels/6d/29/6c/82214c85262c5bdb1d615291039f4ec9f974034103e7a303fa 2025-10-10T01:32:19.1035136Z Successfully built lumen-ci 2025-10-10T01:32:19.2631156Z Installing collected packages: uv, smmap, gitdb, docker, GitPython, lumen-ci 2025-10-10T01:32:19.4312066Z [?25l 2025-10-10T01:32:19.5990251Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/6 [uv] 2025-10-10T01:32:19.6900118Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 [GitPython] 2025-10-10T01:32:19.6900835Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6/6 [lumen-ci] 2025-10-10T01:32:19.6901188Z [?25h 2025-10-10T01:32:19.6993017Z Successfully installed GitPython-3.1.45 docker-7.1.0 gitdb-4.0.12 lumen-ci-0.1.0 smmap-5.0.2 uv-0.8.6 2025-10-10T01:32:19.7880823Z + python -m cli.run test external vllm --test-plan vllm_basic_models_test --shard-id 1 --num-shards 1 2025-10-10T01:32:19.9913344Z 2025-10-10 01:32:19,990 [INFO] cli.lib.core.vllm.vllm_test: Display VllmTestParameters VllmTestParameters(torch_whls_path=PosixPath('/var/lib/jenkins/workspace/dist'), vllm_whls_path=PosixPath('/var/lib/jenkins/workspace/dist/external/vllm/wheels'), torch_cuda_arch_list='8.9', cleaning_script=PosixPath('/var/lib/jenkins/workspace/.github/ci_configs/vllm/use_existing_torch.py')) 2025-10-10T01:32:19.9914987Z 2025-10-10 01:32:19,990 [INFO] cli.lib.common.git_helper: Cloning vllm to vllm 2025-10-10T01:32:20.3126353Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 10% - remote: Counting objects: 10% (7/64) 2025-10-10T01:32:20.3127096Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 15% - remote: Counting objects: 15% (10/64) 2025-10-10T01:32:20.3127755Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 20% - remote: Counting objects: 20% (13/64) 2025-10-10T01:32:20.3128285Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 25% - remote: Counting objects: 25% (16/64) 2025-10-10T01:32:20.3128974Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 35% - remote: Counting objects: 35% (23/64) 2025-10-10T01:32:20.3129614Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 40% - remote: Counting objects: 40% (26/64) 2025-10-10T01:32:20.3130247Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 45% - remote: Counting objects: 45% (29/64) 2025-10-10T01:32:20.3130804Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 50% - remote: Counting objects: 50% (32/64) 2025-10-10T01:32:20.3131343Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 60% - remote: Counting objects: 60% (39/64) 2025-10-10T01:32:20.3132065Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 65% - remote: Counting objects: 65% (42/64) 2025-10-10T01:32:20.3132559Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 70% - remote: Counting objects: 70% (45/64) 2025-10-10T01:32:20.3133541Z 2025-10-10 01:32:20,312 [INFO] cli.lib.common.git_helper: Progress: 75% - remote: Counting objects: 75% (48/64) 2025-10-10T01:32:20.3134162Z 2025-10-10 01:32:20,313 [INFO] cli.lib.common.git_helper: Progress: 85% - remote: Counting objects: 85% (55/64) 2025-10-10T01:32:20.3134658Z 2025-10-10 01:32:20,313 [INFO] cli.lib.common.git_helper: Progress: 90% - remote: Counting objects: 90% (58/64) 2025-10-10T01:32:20.3135146Z 2025-10-10 01:32:20,313 [INFO] cli.lib.common.git_helper: Progress: 95% - remote: Counting objects: 95% (61/64) 2025-10-10T01:32:20.3135654Z 2025-10-10 01:32:20,313 [INFO] cli.lib.common.git_helper: Progress: 100% - remote: Counting objects: 100% (64/64) 2025-10-10T01:32:20.3136166Z 2025-10-10 01:32:20,313 [INFO] cli.lib.common.git_helper: Progress: 5% - remote: Compressing objects: 5% (3/54) 2025-10-10T01:32:20.3248449Z 2025-10-10 01:32:20,324 [INFO] cli.lib.common.git_helper: Progress: 20% - remote: Compressing objects: 20% (11/54) 2025-10-10T01:32:20.3276743Z 2025-10-10 01:32:20,327 [INFO] cli.lib.common.git_helper: Progress: 25% - remote: Compressing objects: 25% (14/54) 2025-10-10T01:32:20.3292657Z 2025-10-10 01:32:20,328 [INFO] cli.lib.common.git_helper: Progress: 35% - remote: Compressing objects: 35% (19/54) 2025-10-10T01:32:20.3293373Z 2025-10-10 01:32:20,329 [INFO] cli.lib.common.git_helper: Progress: 40% - remote: Compressing objects: 40% (22/54) 2025-10-10T01:32:20.3294766Z 2025-10-10 01:32:20,329 [INFO] cli.lib.common.git_helper: Progress: 50% - remote: Compressing objects: 50% (27/54) 2025-10-10T01:32:20.3295574Z 2025-10-10 01:32:20,329 [INFO] cli.lib.common.git_helper: Progress: 55% - remote: Compressing objects: 55% (30/54) 2025-10-10T01:32:20.3297860Z 2025-10-10 01:32:20,329 [INFO] cli.lib.common.git_helper: Progress: 70% - remote: Compressing objects: 70% (38/54) 2025-10-10T01:32:20.3299439Z 2025-10-10 01:32:20,329 [INFO] cli.lib.common.git_helper: Progress: 75% - remote: Compressing objects: 75% (41/54) 2025-10-10T01:32:20.3300120Z 2025-10-10 01:32:20,329 [INFO] cli.lib.common.git_helper: Progress: 85% - remote: Compressing objects: 85% (46/54) 2025-10-10T01:32:20.3302215Z 2025-10-10 01:32:20,330 [INFO] cli.lib.common.git_helper: Progress: 90% - remote: Compressing objects: 90% (49/54) 2025-10-10T01:32:20.3302780Z 2025-10-10 01:32:20,330 [INFO] cli.lib.common.git_helper: Progress: 100% - remote: Compressing objects: 100% (54/54) 2025-10-10T01:32:20.3537179Z 2025-10-10 01:32:20,353 [INFO] cli.lib.common.git_helper: Progress: 0% - Receiving objects: 0% (1/126974) 2025-10-10T01:32:20.4452264Z 2025-10-10 01:32:20,444 [INFO] cli.lib.common.git_helper: Progress: 5% - Receiving objects: 5% (6349/126974) 2025-10-10T01:32:20.5957346Z 2025-10-10 01:32:20,595 [INFO] cli.lib.common.git_helper: Progress: 10% - Receiving objects: 10% (12698/126974) 2025-10-10T01:32:20.8464103Z 2025-10-10 01:32:20,846 [INFO] cli.lib.common.git_helper: Progress: 15% - Receiving objects: 15% (19047/126974) 2025-10-10T01:32:20.9886831Z 2025-10-10 01:32:20,988 [INFO] cli.lib.common.git_helper: Progress: 20% - Receiving objects: 20% (25395/126974), 27.56 MiB | 55.11 MiB/s 2025-10-10T01:32:21.1271525Z 2025-10-10 01:32:21,126 [INFO] cli.lib.common.git_helper: Progress: 25% - Receiving objects: 25% (31744/126974), 27.56 MiB | 55.11 MiB/s 2025-10-10T01:32:21.2305797Z 2025-10-10 01:32:21,230 [INFO] cli.lib.common.git_helper: Progress: 30% - Receiving objects: 30% (38093/126974), 27.56 MiB | 55.11 MiB/s 2025-10-10T01:32:21.3265624Z 2025-10-10 01:32:21,326 [INFO] cli.lib.common.git_helper: Progress: 35% - Receiving objects: 35% (44441/126974), 27.56 MiB | 55.11 MiB/s 2025-10-10T01:32:21.4248209Z 2025-10-10 01:32:21,424 [INFO] cli.lib.common.git_helper: Progress: 40% - Receiving objects: 40% (50790/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.4601247Z 2025-10-10 01:32:21,459 [INFO] cli.lib.common.git_helper: Progress: 45% - Receiving objects: 45% (57139/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.4970259Z 2025-10-10 01:32:21,496 [INFO] cli.lib.common.git_helper: Progress: 50% - Receiving objects: 50% (63487/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.5359707Z 2025-10-10 01:32:21,535 [INFO] cli.lib.common.git_helper: Progress: 55% - Receiving objects: 55% (69836/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.5977179Z 2025-10-10 01:32:21,597 [INFO] cli.lib.common.git_helper: Progress: 60% - Receiving objects: 60% (76185/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.6685114Z 2025-10-10 01:32:21,668 [INFO] cli.lib.common.git_helper: Progress: 65% - Receiving objects: 65% (82534/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.7318035Z 2025-10-10 01:32:21,731 [INFO] cli.lib.common.git_helper: Progress: 70% - Receiving objects: 70% (88882/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.7777187Z 2025-10-10 01:32:21,777 [INFO] cli.lib.common.git_helper: Progress: 75% - Receiving objects: 75% (95231/126974), 55.96 MiB | 55.95 MiB/s 2025-10-10T01:32:21.8691426Z 2025-10-10 01:32:21,868 [INFO] cli.lib.common.git_helper: Progress: 80% - Receiving objects: 80% (101580/126974), 85.96 MiB | 57.30 MiB/s 2025-10-10T01:32:21.9121680Z 2025-10-10 01:32:21,911 [INFO] cli.lib.common.git_helper: Progress: 85% - Receiving objects: 85% (107928/126974), 85.96 MiB | 57.30 MiB/s 2025-10-10T01:32:21.9451753Z 2025-10-10 01:32:21,944 [INFO] cli.lib.common.git_helper: Progress: 90% - Receiving objects: 90% (114277/126974), 85.96 MiB | 57.30 MiB/s 2025-10-10T01:32:21.9917808Z 2025-10-10 01:32:21,991 [INFO] cli.lib.common.git_helper: Progress: 95% - Receiving objects: 95% (120626/126974), 85.96 MiB | 57.30 MiB/s 2025-10-10T01:32:22.0241653Z 2025-10-10 01:32:22,023 [INFO] cli.lib.common.git_helper: Progress: 100% - Receiving objects: 100% (126974/126974), 85.96 MiB | 57.30 MiB/s 2025-10-10T01:32:22.0419958Z 2025-10-10 01:32:22,041 [INFO] cli.lib.common.git_helper: Resolving deltas: 0% (0/100681) 2025-10-10T01:32:22.0746303Z 2025-10-10 01:32:22,074 [INFO] cli.lib.common.git_helper: Progress: 5% - Resolving deltas: 5% (5035/100681) 2025-10-10T01:32:22.1252437Z 2025-10-10 01:32:22,124 [INFO] cli.lib.common.git_helper: Progress: 10% - Resolving deltas: 10% (10069/100681) 2025-10-10T01:32:22.1723111Z 2025-10-10 01:32:22,171 [INFO] cli.lib.common.git_helper: Progress: 15% - Resolving deltas: 15% (15103/100681) 2025-10-10T01:32:22.2229361Z 2025-10-10 01:32:22,222 [INFO] cli.lib.common.git_helper: Progress: 20% - Resolving deltas: 20% (20137/100681) 2025-10-10T01:32:22.2726639Z 2025-10-10 01:32:22,272 [INFO] cli.lib.common.git_helper: Progress: 25% - Resolving deltas: 25% (25171/100681) 2025-10-10T01:32:22.3178523Z 2025-10-10 01:32:22,317 [INFO] cli.lib.common.git_helper: Progress: 30% - Resolving deltas: 30% (30205/100681) 2025-10-10T01:32:22.3550903Z 2025-10-10 01:32:22,354 [INFO] cli.lib.common.git_helper: Progress: 35% - Resolving deltas: 35% (35239/100681) 2025-10-10T01:32:22.3890354Z 2025-10-10 01:32:22,388 [INFO] cli.lib.common.git_helper: Progress: 40% - Resolving deltas: 40% (40273/100681) 2025-10-10T01:32:22.4272605Z 2025-10-10 01:32:22,426 [INFO] cli.lib.common.git_helper: Progress: 45% - Resolving deltas: 45% (45307/100681) 2025-10-10T01:32:22.4479768Z 2025-10-10 01:32:22,447 [INFO] cli.lib.common.git_helper: Progress: 50% - Resolving deltas: 50% (50341/100681) 2025-10-10T01:32:22.4680653Z 2025-10-10 01:32:22,467 [INFO] cli.lib.common.git_helper: Progress: 55% - Resolving deltas: 55% (55375/100681) 2025-10-10T01:32:22.4893022Z 2025-10-10 01:32:22,488 [INFO] cli.lib.common.git_helper: Progress: 60% - Resolving deltas: 60% (60409/100681) 2025-10-10T01:32:22.5110761Z 2025-10-10 01:32:22,510 [INFO] cli.lib.common.git_helper: Progress: 65% - Resolving deltas: 65% (65445/100681) 2025-10-10T01:32:22.5396626Z 2025-10-10 01:32:22,539 [INFO] cli.lib.common.git_helper: Progress: 70% - Resolving deltas: 70% (70477/100681) 2025-10-10T01:32:22.5597331Z 2025-10-10 01:32:22,559 [INFO] cli.lib.common.git_helper: Progress: 75% - Resolving deltas: 75% (75511/100681) 2025-10-10T01:32:22.5807491Z 2025-10-10 01:32:22,580 [INFO] cli.lib.common.git_helper: Progress: 80% - Resolving deltas: 80% (80545/100681) 2025-10-10T01:32:22.6085663Z 2025-10-10 01:32:22,608 [INFO] cli.lib.common.git_helper: Progress: 85% - Resolving deltas: 85% (85579/100681) 2025-10-10T01:32:22.6271969Z 2025-10-10 01:32:22,626 [INFO] cli.lib.common.git_helper: Progress: 90% - Resolving deltas: 90% (90613/100681) 2025-10-10T01:32:22.6442660Z 2025-10-10 01:32:22,643 [INFO] cli.lib.common.git_helper: Progress: 95% - Resolving deltas: 95% (95647/100681) 2025-10-10T01:32:22.6555789Z 2025-10-10 01:32:22,655 [INFO] cli.lib.common.git_helper: Progress: 100% - Resolving deltas: 100% (100681/100681) 2025-10-10T01:32:23.3713112Z 2025-10-10 01:32:23,370 [INFO] cli.lib.common.git_helper: Checking out pinned vllm commit 0ad9951c416d33c5da4f7a504fb162cbe62386f5 2025-10-10T01:32:23.6965733Z 2025-10-10 01:32:23,695 [INFO] cli.lib.common.git_helper: Successfully cloned vllm 2025-10-10T01:32:23.7341858Z 2025-10-10 01:32:23,733 [INFO] cli.lib.core.vllm.vllm_test: Running vllm test with inputs: VllmTestParameters(torch_whls_path=PosixPath('/var/lib/jenkins/workspace/dist'), vllm_whls_path=PosixPath('/var/lib/jenkins/workspace/dist/external/vllm/wheels'), torch_cuda_arch_list='8.9', cleaning_script=PosixPath('/var/lib/jenkins/workspace/.github/ci_configs/vllm/use_existing_torch.py')) 2025-10-10T01:32:23.7360897Z 2025-10-10 01:32:23,735 [INFO] cli.lib.common.pip_helper: torch already exist with version: 2.10.0a0+git344e636 2025-10-10T01:32:23.7363315Z 2025-10-10 01:32:23,736 [INFO] cli.lib.common.pip_helper: Installing /var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0+966da7e-cp312-cp312-linux_x86_64.whl... 2025-10-10T01:32:23.7364712Z 2025-10-10 01:32:23,736 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0+966da7e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:32:23.7366156Z 2025-10-10 01:32:23,736 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0+966da7e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:32:24.0267968Z Processing /var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0+966da7e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:32:24.0343283Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torchvision==0.22.0a0+966da7e) (1.26.2) 2025-10-10T01:32:24.0346141Z Requirement already satisfied: torch in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torchvision==0.22.0a0+966da7e) (2.10.0a0+git344e636) 2025-10-10T01:32:24.0349739Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torchvision==0.22.0a0+966da7e) (11.0.0) 2025-10-10T01:32:24.0402783Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (3.18.0) 2025-10-10T01:32:24.0406602Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (4.12.2) 2025-10-10T01:32:24.0411537Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (78.1.1) 2025-10-10T01:32:24.0415234Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (1.13.3) 2025-10-10T01:32:24.0418905Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (2.8.8) 2025-10-10T01:32:24.0426278Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (3.1.6) 2025-10-10T01:32:24.0430103Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchvision==0.22.0a0+966da7e) (2025.9.0) 2025-10-10T01:32:24.0530660Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch->torchvision==0.22.0a0+966da7e) (1.3.0) 2025-10-10T01:32:24.0570059Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2->torch->torchvision==0.22.0a0+966da7e) (3.0.3) 2025-10-10T01:32:24.2180162Z Installing collected packages: torchvision 2025-10-10T01:32:24.6406941Z Successfully installed torchvision-0.22.0a0+966da7e 2025-10-10T01:32:24.6842069Z 2025-10-10 01:32:24,683 [INFO] cli.lib.common.pip_helper: Installing /var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0+87ff22e-cp312-cp312-linux_x86_64.whl... 2025-10-10T01:32:24.6843489Z 2025-10-10 01:32:24,683 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0+87ff22e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:32:24.6845005Z 2025-10-10 01:32:24,683 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0+87ff22e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:32:24.9764016Z Processing /var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0+87ff22e-cp312-cp312-linux_x86_64.whl 2025-10-10T01:32:24.9820817Z Requirement already satisfied: torch in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torchaudio==2.8.0a0+87ff22e) (2.10.0a0+git344e636) 2025-10-10T01:32:24.9829784Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (3.18.0) 2025-10-10T01:32:24.9834000Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (4.12.2) 2025-10-10T01:32:24.9839595Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (78.1.1) 2025-10-10T01:32:24.9843741Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (1.13.3) 2025-10-10T01:32:24.9847489Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (2.8.8) 2025-10-10T01:32:24.9850443Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (3.1.6) 2025-10-10T01:32:24.9854308Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->torchaudio==2.8.0a0+87ff22e) (2025.9.0) 2025-10-10T01:32:24.9961320Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch->torchaudio==2.8.0a0+87ff22e) (1.3.0) 2025-10-10T01:32:25.0002102Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2->torch->torchaudio==2.8.0a0+87ff22e) (3.0.3) 2025-10-10T01:32:25.1718912Z Installing collected packages: torchaudio 2025-10-10T01:32:25.3225850Z Successfully installed torchaudio-2.8.0a0+87ff22e 2025-10-10T01:32:25.3648333Z 2025-10-10 01:32:25,364 [INFO] cli.lib.core.vllm.vllm_test: Done. Installed torch and other torch-related wheels 2025-10-10T01:32:25.3649100Z 2025-10-10 01:32:25,364 [INFO] cli.lib.core.vllm.vllm_test: Installing vllm wheels 2025-10-10T01:32:25.3650599Z 2025-10-10 01:32:25,364 [INFO] cli.lib.common.pip_helper: Installing /var/lib/jenkins/workspace/dist/external/vllm/wheels/xformers/xformers-0.0.33+5d4b92a5.d20251010-cp39-abi3-linux_x86_64.whl... 2025-10-10T01:32:25.3652425Z 2025-10-10 01:32:25,364 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/external/vllm/wheels/xformers/xformers-0.0.33+5d4b92a5.d20251010-cp39-abi3-linux_x86_64.whl 2025-10-10T01:32:25.3654634Z 2025-10-10 01:32:25,364 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/external/vllm/wheels/xformers/xformers-0.0.33+5d4b92a5.d20251010-cp39-abi3-linux_x86_64.whl 2025-10-10T01:32:25.6572932Z Processing /var/lib/jenkins/workspace/dist/external/vllm/wheels/xformers/xformers-0.0.33+5d4b92a5.d20251010-cp39-abi3-linux_x86_64.whl 2025-10-10T01:32:25.8466466Z Requirement already satisfied: torch>=2.8 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from xformers==0.0.33+5d4b92a5.d20251010) (2.10.0a0+git344e636) 2025-10-10T01:32:25.8468506Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from xformers==0.0.33+5d4b92a5.d20251010) (1.26.2) 2025-10-10T01:32:25.8478348Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (3.18.0) 2025-10-10T01:32:25.8483172Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (4.12.2) 2025-10-10T01:32:25.8487804Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (78.1.1) 2025-10-10T01:32:25.8491568Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (1.13.3) 2025-10-10T01:32:25.8495902Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (2.8.8) 2025-10-10T01:32:25.8498904Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (3.1.6) 2025-10-10T01:32:25.8502586Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (2025.9.0) 2025-10-10T01:32:25.8609818Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (1.3.0) 2025-10-10T01:32:25.8649398Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2->torch>=2.8->xformers==0.0.33+5d4b92a5.d20251010) (3.0.3) 2025-10-10T01:32:26.0308390Z Installing collected packages: xformers 2025-10-10T01:32:29.7000957Z Successfully installed xformers-0.0.33+5d4b92a5.d20251010 2025-10-10T01:32:29.7436762Z 2025-10-10 01:32:29,743 [INFO] cli.lib.common.pip_helper: Installing /var/lib/jenkins/workspace/dist/external/vllm/wheels/vllm/vllm-0.11.0rc2.dev157+g0ad9951c4.d20251010-cp38-abi3-linux_x86_64.whl... 2025-10-10T01:32:29.7438381Z 2025-10-10 01:32:29,743 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/external/vllm/wheels/vllm/vllm-0.11.0rc2.dev157+g0ad9951c4.d20251010-cp38-abi3-linux_x86_64.whl 2025-10-10T01:32:29.7439907Z 2025-10-10 01:32:29,743 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/external/vllm/wheels/vllm/vllm-0.11.0rc2.dev157+g0ad9951c4.d20251010-cp38-abi3-linux_x86_64.whl 2025-10-10T01:32:30.0340227Z Processing /var/lib/jenkins/workspace/dist/external/vllm/wheels/vllm/vllm-0.11.0rc2.dev157+g0ad9951c4.d20251010-cp38-abi3-linux_x86_64.whl 2025-10-10T01:32:30.5815292Z Collecting regex (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:30.6007991Z Downloading regex-2025.9.18-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB) 2025-10-10T01:32:30.6087453Z Requirement already satisfied: cachetools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (6.2.0) 2025-10-10T01:32:30.6090529Z Requirement already satisfied: psutil in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (7.1.0) 2025-10-10T01:32:30.6456668Z Collecting sentencepiece (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:30.6497515Z Downloading sentencepiece-0.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (10 kB) 2025-10-10T01:32:30.6553068Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.26.2) 2025-10-10T01:32:30.6556314Z Requirement already satisfied: requests>=2.26.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.32.5) 2025-10-10T01:32:30.6558746Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (4.67.1) 2025-10-10T01:32:30.6985560Z Collecting blake3 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:30.7014003Z Downloading blake3-1.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (217 bytes) 2025-10-10T01:32:30.7139861Z Collecting py-cpuinfo (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:30.7180657Z Downloading py_cpuinfo-9.0.0-py3-none-any.whl.metadata (794 bytes) 2025-10-10T01:32:30.7580928Z Collecting transformers>=4.55.2 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:30.7618492Z Downloading transformers-4.57.0-py3-none-any.whl.metadata (41 kB) 2025-10-10T01:32:30.9557568Z Collecting tokenizers>=0.21.1 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:30.9588047Z Downloading tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB) 2025-10-10T01:32:30.9644785Z Requirement already satisfied: protobuf in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (5.29.5) 2025-10-10T01:32:30.9976803Z Collecting fastapi>=0.115.0 (from fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.0006976Z Downloading fastapi-0.118.2-py3-none-any.whl.metadata (28 kB) 2025-10-10T01:32:31.0099031Z Requirement already satisfied: aiohttp in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.13.0) 2025-10-10T01:32:31.0714311Z Collecting openai>=1.99.1 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.0742828Z Downloading openai-2.3.0-py3-none-any.whl.metadata (29 kB) 2025-10-10T01:32:31.1645677Z Collecting pydantic>=2.11.7 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.1672546Z Downloading pydantic-2.12.0-py3-none-any.whl.metadata (83 kB) 2025-10-10T01:32:31.1889408Z Collecting prometheus_client>=0.18.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.1923916Z Downloading prometheus_client-0.23.1-py3-none-any.whl.metadata (1.9 kB) 2025-10-10T01:32:31.1969858Z Requirement already satisfied: pillow in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (11.0.0) 2025-10-10T01:32:31.2110638Z Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.2145171Z Downloading prometheus_fastapi_instrumentator-7.1.0-py3-none-any.whl.metadata (13 kB) 2025-10-10T01:32:31.2504802Z Collecting tiktoken>=0.6.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.2543005Z Downloading tiktoken-0.12.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (6.7 kB) 2025-10-10T01:32:31.2786155Z Collecting lm-format-enforcer==0.11.3 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.2814672Z Downloading lm_format_enforcer-0.11.3-py3-none-any.whl.metadata (17 kB) 2025-10-10T01:32:31.3245458Z Collecting llguidance<0.8.0,>=0.7.11 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.3283153Z Downloading llguidance-0.7.30-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB) 2025-10-10T01:32:31.3558592Z Collecting outlines_core==0.2.11 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.3593642Z Downloading outlines_core-0.2.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.8 kB) 2025-10-10T01:32:31.3817996Z Collecting diskcache==5.6.3 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.3864068Z Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB) 2025-10-10T01:32:31.3997108Z Collecting lark==1.2.2 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.4032812Z Downloading lark-1.2.2-py3-none-any.whl.metadata (1.8 kB) 2025-10-10T01:32:31.4410141Z Collecting xgrammar==0.1.25 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.4440749Z Downloading xgrammar-0.1.25-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB) 2025-10-10T01:32:31.4500553Z Requirement already satisfied: typing_extensions>=4.10 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (4.12.2) 2025-10-10T01:32:31.4503198Z Requirement already satisfied: filelock>=3.16.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.18.0) 2025-10-10T01:32:31.4636015Z Collecting partial-json-parser (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.4693611Z Downloading partial_json_parser-0.2.1.1.post6-py3-none-any.whl.metadata (6.1 kB) 2025-10-10T01:32:31.6604494Z Collecting pyzmq>=25.0.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.6633579Z Downloading pyzmq-27.1.0-cp312-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl.metadata (6.0 kB) 2025-10-10T01:32:31.7074021Z Collecting msgspec (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.7110588Z Downloading msgspec-0.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.9 kB) 2025-10-10T01:32:31.7284233Z Collecting gguf>=0.13.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.7312237Z Downloading gguf-0.17.1-py3-none-any.whl.metadata (4.3 kB) 2025-10-10T01:32:31.7468914Z Collecting mistral_common>=1.8.2 (from mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.7495309Z Downloading mistral_common-1.8.5-py3-none-any.whl.metadata (5.1 kB) 2025-10-10T01:32:31.8296285Z Collecting opencv-python-headless>=4.11.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.8328156Z Downloading opencv_python_headless-4.12.0.88-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (19 kB) 2025-10-10T01:32:31.8382760Z Requirement already satisfied: pyyaml in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (6.0.2) 2025-10-10T01:32:31.8387721Z Requirement already satisfied: six>=1.16.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.17.0) 2025-10-10T01:32:31.8392523Z Requirement already satisfied: setuptools<80,>=77.0.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (78.1.1) 2025-10-10T01:32:31.8494522Z Collecting einops (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.8543251Z Downloading einops-0.8.1-py3-none-any.whl.metadata (13 kB) 2025-10-10T01:32:31.8775720Z Collecting compressed-tensors==0.11.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.8812771Z Downloading compressed_tensors-0.11.0-py3-none-any.whl.metadata (7.0 kB) 2025-10-10T01:32:31.8967512Z Collecting depyf==0.19.0 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.8998438Z Downloading depyf-0.19.0-py3-none-any.whl.metadata (7.3 kB) 2025-10-10T01:32:31.9169598Z Collecting cloudpickle (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.9204956Z Downloading cloudpickle-3.1.1-py3-none-any.whl.metadata (7.1 kB) 2025-10-10T01:32:31.9772554Z Collecting watchfiles (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:31.9814437Z Downloading watchfiles-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB) 2025-10-10T01:32:31.9989134Z Collecting python-json-logger (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.0024716Z Downloading python_json_logger-4.0.0-py3-none-any.whl.metadata (4.0 kB) 2025-10-10T01:32:32.0084917Z Requirement already satisfied: scipy in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.14.1) 2025-10-10T01:32:32.0087618Z Requirement already satisfied: ninja in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.10.2) 2025-10-10T01:32:32.0790935Z Collecting pybase64 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.0825435Z Downloading pybase64-1.4.2-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl.metadata (8.7 kB) 2025-10-10T01:32:32.1185454Z Collecting cbor2 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.1245299Z Downloading cbor2-5.7.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (5.4 kB) 2025-10-10T01:32:32.1669674Z Collecting setproctitle (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.1704590Z Downloading setproctitle-1.3.7-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (10 kB) 2025-10-10T01:32:32.1981805Z Collecting openai-harmony>=0.0.3 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.2014737Z Downloading openai_harmony-0.0.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.0 kB) 2025-10-10T01:32:32.2748135Z Collecting numba==0.61.2 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.2784603Z Downloading numba-0.61.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.8 kB) 2025-10-10T01:32:32.3658819Z Collecting ray>=2.48.0 (from ray[cgraph]>=2.48.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.3707139Z Downloading ray-2.49.2-cp312-cp312-manylinux2014_x86_64.whl.metadata (21 kB) 2025-10-10T01:32:32.4228512Z Requirement already satisfied: torch>=1.7.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from compressed-tensors==0.11.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.10.0a0+git344e636) 2025-10-10T01:32:32.4599509Z Collecting frozendict (from compressed-tensors==0.11.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.4646631Z Downloading frozendict-2.4.6-py312-none-any.whl.metadata (23 kB) 2025-10-10T01:32:32.4814748Z Collecting astor (from depyf==0.19.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.4846563Z Downloading astor-0.8.1-py2.py3-none-any.whl.metadata (4.2 kB) 2025-10-10T01:32:32.4902916Z Requirement already satisfied: dill in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from depyf==0.19.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (0.3.7) 2025-10-10T01:32:32.5033666Z Collecting interegular>=0.3.2 (from lm-format-enforcer==0.11.3->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.5068956Z Downloading interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB) 2025-10-10T01:32:32.5125428Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from lm-format-enforcer==0.11.3->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (23.1) 2025-10-10T01:32:32.5642199Z Collecting llvmlite<0.45,>=0.44.0dev0 (from numba==0.61.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.6087621Z Downloading llvmlite-0.44.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.0 kB) 2025-10-10T01:32:32.6231811Z Requirement already satisfied: triton in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from xgrammar==0.1.25->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.5.0+git7416ffcb) 2025-10-10T01:32:32.6560080Z Collecting starlette<0.49.0,>=0.40.0 (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.6595657Z Downloading starlette-0.48.0-py3-none-any.whl.metadata (6.3 kB) 2025-10-10T01:32:32.6933800Z Collecting annotated-types>=0.6.0 (from pydantic>=2.11.7->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:32.6970154Z Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB) 2025-10-10T01:32:33.2414043Z Collecting pydantic-core==2.41.1 (from pydantic>=2.11.7->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.2445734Z Downloading pydantic_core-2.41.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB) 2025-10-10T01:32:33.2619706Z Collecting typing_extensions>=4.10 (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.2656242Z Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) 2025-10-10T01:32:33.2783982Z Collecting typing-inspection>=0.4.2 (from pydantic>=2.11.7->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.2812284Z Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB) 2025-10-10T01:32:33.3054851Z Collecting anyio<5,>=3.6.2 (from starlette<0.49.0,>=0.40.0->fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.3092137Z Downloading anyio-4.11.0-py3-none-any.whl.metadata (4.1 kB) 2025-10-10T01:32:33.3152080Z Requirement already satisfied: idna>=2.8 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from anyio<5,>=3.6.2->starlette<0.49.0,>=0.40.0->fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.10) 2025-10-10T01:32:33.3242328Z Collecting sniffio>=1.1 (from anyio<5,>=3.6.2->starlette<0.49.0,>=0.40.0->fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.3320606Z Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB) 2025-10-10T01:32:33.3489782Z Collecting fastapi-cli>=0.0.8 (from fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.3539722Z Downloading fastapi_cli-0.0.13-py3-none-any.whl.metadata (6.3 kB) 2025-10-10T01:32:33.3736042Z Collecting httpx<1.0.0,>=0.23.0 (from fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.3767630Z Downloading httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB) 2025-10-10T01:32:33.3827700Z Requirement already satisfied: jinja2>=3.1.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.1.6) 2025-10-10T01:32:33.4064794Z Collecting python-multipart>=0.0.18 (from fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.4093009Z Downloading python_multipart-0.0.20-py3-none-any.whl.metadata (1.8 kB) 2025-10-10T01:32:33.4313381Z Collecting email-validator>=2.0.0 (from fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.4373331Z Downloading email_validator-2.3.0-py3-none-any.whl.metadata (26 kB) 2025-10-10T01:32:33.4659346Z Collecting uvicorn>=0.12.0 (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.4693671Z Downloading uvicorn-0.37.0-py3-none-any.whl.metadata (6.6 kB) 2025-10-10T01:32:33.4762493Z Requirement already satisfied: certifi in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2025.10.5) 2025-10-10T01:32:33.4916378Z Collecting httpcore==1.* (from httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.4944802Z Downloading httpcore-1.0.9-py3-none-any.whl.metadata (21 kB) 2025-10-10T01:32:33.5108432Z Collecting h11>=0.16 (from httpcore==1.*->httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.5142745Z Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB) 2025-10-10T01:32:33.5303173Z Collecting dnspython>=2.0.0 (from email-validator>=2.0.0->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.5329889Z Downloading dnspython-2.8.0-py3-none-any.whl.metadata (5.7 kB) 2025-10-10T01:32:33.5553035Z Collecting typer>=0.15.1 (from fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.5580181Z Downloading typer-0.19.2-py3-none-any.whl.metadata (16 kB) 2025-10-10T01:32:33.5851013Z Collecting rich-toolkit>=0.14.8 (from fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.5887226Z Downloading rich_toolkit-0.15.1-py3-none-any.whl.metadata (1.0 kB) 2025-10-10T01:32:33.6066147Z Collecting fastapi-cloud-cli>=0.1.1 (from fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.6093720Z Downloading fastapi_cloud_cli-0.3.1-py3-none-any.whl.metadata (3.2 kB) 2025-10-10T01:32:33.6776720Z Collecting rignore>=0.5.1 (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.6804788Z Downloading rignore-0.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB) 2025-10-10T01:32:33.7485627Z Collecting sentry-sdk>=2.20.0 (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.7513358Z Downloading sentry_sdk-2.41.0-py2.py3-none-any.whl.metadata (10 kB) 2025-10-10T01:32:33.7738497Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2>=3.1.5->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.0.3) 2025-10-10T01:32:33.8151468Z Collecting jsonschema>=4.21.1 (from mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.8179239Z Downloading jsonschema-4.25.1-py3-none-any.whl.metadata (7.6 kB) 2025-10-10T01:32:33.8371635Z Collecting pydantic-extra-types>=2.10.5 (from pydantic-extra-types[pycountry]>=2.10.5->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.8402472Z Downloading pydantic_extra_types-2.10.6-py3-none-any.whl.metadata (4.0 kB) 2025-10-10T01:32:33.8492228Z Requirement already satisfied: attrs>=22.2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (25.4.0) 2025-10-10T01:32:33.8607320Z Collecting jsonschema-specifications>=2023.03.6 (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.8655541Z Downloading jsonschema_specifications-2025.9.1-py3-none-any.whl.metadata (2.9 kB) 2025-10-10T01:32:33.8884021Z Collecting referencing>=0.28.4 (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:33.8910885Z Downloading referencing-0.36.2-py3-none-any.whl.metadata (2.8 kB) 2025-10-10T01:32:34.0929546Z Collecting rpds-py>=0.7.1 (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.0967155Z Downloading rpds_py-0.27.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB) 2025-10-10T01:32:34.1163649Z Requirement already satisfied: distro<2,>=1.7.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from openai>=1.99.1->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.9.0) 2025-10-10T01:32:34.1656793Z Collecting jiter<1,>=0.10.0 (from openai>=1.99.1->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.1688538Z Downloading jiter-0.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.2 kB) 2025-10-10T01:32:34.4056953Z Collecting numpy (from vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.4096724Z Downloading numpy-2.2.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB) 2025-10-10T01:32:34.4776847Z Collecting pycountry>=23 (from pydantic-extra-types[pycountry]>=2.10.5->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.4816208Z Downloading pycountry-24.6.1-py3-none-any.whl.metadata (12 kB) 2025-10-10T01:32:34.4950751Z Requirement already satisfied: click>=7.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from ray>=2.48.0->ray[cgraph]>=2.48.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (8.3.0) 2025-10-10T01:32:34.4954959Z Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from ray>=2.48.0->ray[cgraph]>=2.48.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.1.2) 2025-10-10T01:32:34.5243064Z Collecting cupy-cuda12x (from ray[cgraph]>=2.48.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.5270177Z Downloading cupy_cuda12x-13.6.0-cp312-cp312-manylinux2014_x86_64.whl.metadata (2.4 kB) 2025-10-10T01:32:34.5460260Z Requirement already satisfied: charset_normalizer<4,>=2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests>=2.26.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (3.4.3) 2025-10-10T01:32:34.5465990Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests>=2.26.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.5.0) 2025-10-10T01:32:34.5828178Z Collecting rich>=13.7.1 (from rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.5899197Z Downloading rich-14.2.0-py3-none-any.whl.metadata (18 kB) 2025-10-10T01:32:34.6142269Z Collecting markdown-it-py>=2.2.0 (from rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.6177484Z Downloading markdown_it_py-4.0.0-py3-none-any.whl.metadata (7.3 kB) 2025-10-10T01:32:34.6253882Z Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.15.0) 2025-10-10T01:32:34.6368847Z Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.6408249Z Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB) 2025-10-10T01:32:34.7026427Z Collecting huggingface-hub<2.0,>=0.16.4 (from tokenizers>=0.21.1->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.7058702Z Downloading huggingface_hub-0.35.3-py3-none-any.whl.metadata (14 kB) 2025-10-10T01:32:34.7276230Z Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.21.1->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2025.9.0) 2025-10-10T01:32:34.7578334Z Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.21.1->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.7617951Z Downloading hf_xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.7 kB) 2025-10-10T01:32:34.7794998Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=1.7.0->compressed-tensors==0.11.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.13.3) 2025-10-10T01:32:34.7799200Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch>=1.7.0->compressed-tensors==0.11.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.8.8) 2025-10-10T01:32:34.7852952Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=1.7.0->compressed-tensors==0.11.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.3.0) 2025-10-10T01:32:34.9314961Z Collecting safetensors>=0.4.3 (from transformers>=4.55.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:34.9356130Z Downloading safetensors-0.6.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB) 2025-10-10T01:32:35.0054079Z Collecting shellingham>=1.3.0 (from typer>=0.15.1->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:35.0082646Z Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB) 2025-10-10T01:32:35.0923220Z Collecting httptools>=0.6.3 (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:35.0951210Z Downloading httptools-0.6.4-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB) 2025-10-10T01:32:35.1729941Z Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:35.1759864Z Downloading python_dotenv-1.1.1-py3-none-any.whl.metadata (24 kB) 2025-10-10T01:32:35.2801143Z Collecting uvloop>=0.15.1 (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:35.2829150Z Downloading uvloop-0.21.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB) 2025-10-10T01:32:35.4199981Z Collecting websockets>=10.4 (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:35.4229855Z Downloading websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB) 2025-10-10T01:32:35.4693528Z Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from aiohttp->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.6.1) 2025-10-10T01:32:35.4697491Z Requirement already satisfied: aiosignal>=1.4.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from aiohttp->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.4.0) 2025-10-10T01:32:35.4704957Z Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from aiohttp->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.8.0) 2025-10-10T01:32:35.4709061Z Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from aiohttp->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (6.7.0) 2025-10-10T01:32:35.4712764Z Requirement already satisfied: propcache>=0.2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from aiohttp->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (0.3.2) 2025-10-10T01:32:35.4716632Z Requirement already satisfied: yarl<2.0,>=1.17.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from aiohttp->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.22.0) 2025-10-10T01:32:35.5441876Z Collecting fastrlock>=0.5 (from cupy-cuda12x->ray[cgraph]>=2.48.0->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) 2025-10-10T01:32:35.5470031Z Downloading fastrlock-0.8.3-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl.metadata (7.7 kB) 2025-10-10T01:32:35.5739936Z Requirement already satisfied: soundfile>=0.12.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (0.13.1) 2025-10-10T01:32:35.5759969Z Requirement already satisfied: cffi>=1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from soundfile>=0.12.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.0.0) 2025-10-10T01:32:35.5780152Z Requirement already satisfied: pycparser in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from cffi>=1.0->soundfile>=0.12.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (2.23) 2025-10-10T01:32:35.5807980Z Requirement already satisfied: soxr>=0.5.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.11.0rc2.dev157+g0ad9951c4.d20251010) (1.0.0) 2025-10-10T01:32:35.6329905Z Downloading compressed_tensors-0.11.0-py3-none-any.whl (179 kB) 2025-10-10T01:32:35.6879283Z Downloading depyf-0.19.0-py3-none-any.whl (39 kB) 2025-10-10T01:32:35.7288472Z Downloading diskcache-5.6.3-py3-none-any.whl (45 kB) 2025-10-10T01:32:35.7567507Z Downloading lark-1.2.2-py3-none-any.whl (111 kB) 2025-10-10T01:32:35.7798815Z Downloading lm_format_enforcer-0.11.3-py3-none-any.whl (45 kB) 2025-10-10T01:32:35.7999486Z Downloading numba-0.61.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.9 MB) 2025-10-10T01:32:35.8528834Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/3.9 MB ? eta -:--:-- 2025-10-10T01:32:35.8529590Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.9/3.9 MB 77.6 MB/s 0:00:00 2025-10-10T01:32:35.8610484Z [?25hDownloading outlines_core-0.2.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB) 2025-10-10T01:32:35.8917480Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.3 MB ? eta -:--:-- 2025-10-10T01:32:35.8918106Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 74.6 MB/s 0:00:00 2025-10-10T01:32:35.8998968Z [?25hDownloading xgrammar-0.1.25-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.7 MB) 2025-10-10T01:32:35.9830500Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/8.7 MB ? eta -:--:-- 2025-10-10T01:32:35.9831137Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.7/8.7 MB 105.4 MB/s 0:00:00 2025-10-10T01:32:35.9937880Z [?25hDownloading llguidance-0.7.30-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.0 MB) 2025-10-10T01:32:36.1964972Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/15.0 MB ? eta -:--:-- 2025-10-10T01:32:36.2039325Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 14.9/15.0 MB 114.3 MB/s eta 0:00:01 2025-10-10T01:32:36.2040009Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.0/15.0 MB 71.0 MB/s 0:00:00 2025-10-10T01:32:36.2071826Z [?25hDownloading llvmlite-0.44.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.4 MB) 2025-10-10T01:32:36.4096565Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/42.4 MB ? eta -:--:-- 2025-10-10T01:32:36.5946525Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━ 31.2/42.4 MB 159.2 MB/s eta 0:00:01 2025-10-10T01:32:36.5947225Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.4/42.4 MB 110.6 MB/s 0:00:00 2025-10-10T01:32:36.5981752Z [?25hDownloading fastapi-0.118.2-py3-none-any.whl (97 kB) 2025-10-10T01:32:36.6065089Z Downloading pydantic-2.12.0-py3-none-any.whl (459 kB) 2025-10-10T01:32:36.6164356Z Downloading pydantic_core-2.41.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB) 2025-10-10T01:32:36.6314301Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 MB ? eta -:--:-- 2025-10-10T01:32:36.6314935Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 147.5 MB/s 0:00:00 2025-10-10T01:32:36.6347730Z [?25hDownloading starlette-0.48.0-py3-none-any.whl (73 kB) 2025-10-10T01:32:36.6420576Z Downloading anyio-4.11.0-py3-none-any.whl (109 kB) 2025-10-10T01:32:36.6494784Z Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB) 2025-10-10T01:32:36.6553065Z Downloading httpx-0.28.1-py3-none-any.whl (73 kB) 2025-10-10T01:32:36.6616534Z Downloading httpcore-1.0.9-py3-none-any.whl (78 kB) 2025-10-10T01:32:36.6686500Z Downloading email_validator-2.3.0-py3-none-any.whl (35 kB) 2025-10-10T01:32:36.6744865Z Downloading dnspython-2.8.0-py3-none-any.whl (331 kB) 2025-10-10T01:32:36.6825652Z Downloading fastapi_cli-0.0.13-py3-none-any.whl (11 kB) 2025-10-10T01:32:36.6884864Z Downloading fastapi_cloud_cli-0.3.1-py3-none-any.whl (19 kB) 2025-10-10T01:32:36.6946793Z Downloading gguf-0.17.1-py3-none-any.whl (96 kB) 2025-10-10T01:32:36.7010018Z Downloading h11-0.16.0-py3-none-any.whl (37 kB) 2025-10-10T01:32:36.7078142Z Downloading interegular-0.3.3-py37-none-any.whl (23 kB) 2025-10-10T01:32:36.7136454Z Downloading mistral_common-1.8.5-py3-none-any.whl (6.5 MB) 2025-10-10T01:32:36.7776016Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/6.5 MB ? eta -:--:-- 2025-10-10T01:32:36.7776648Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 102.9 MB/s 0:00:00 2025-10-10T01:32:36.7806947Z [?25hDownloading jsonschema-4.25.1-py3-none-any.whl (90 kB) 2025-10-10T01:32:36.7871521Z Downloading jsonschema_specifications-2025.9.1-py3-none-any.whl (18 kB) 2025-10-10T01:32:36.7930981Z Downloading openai-2.3.0-py3-none-any.whl (999 kB) 2025-10-10T01:32:36.8022806Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/999.8 kB ? eta -:--:-- 2025-10-10T01:32:36.8023446Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 999.8/999.8 kB 109.9 MB/s 0:00:00 2025-10-10T01:32:36.8051883Z [?25hDownloading jiter-0.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (347 kB) 2025-10-10T01:32:36.8130543Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-10-10T01:32:36.8198829Z Downloading openai_harmony-0.0.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB) 2025-10-10T01:32:36.8557454Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/3.0 MB ? eta -:--:-- 2025-10-10T01:32:36.8558373Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 85.0 MB/s 0:00:00 2025-10-10T01:32:36.8587461Z [?25hDownloading opencv_python_headless-4.12.0.88-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (54.0 MB) 2025-10-10T01:32:37.0623306Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/54.0 MB ? eta -:--:-- 2025-10-10T01:32:37.2636993Z  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 21.5/54.0 MB 106.8 MB/s eta 0:00:01 2025-10-10T01:32:37.3994704Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 47.7/54.0 MB 118.7 MB/s eta 0:00:01 2025-10-10T01:32:37.3995418Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 MB 100.1 MB/s 0:00:00 2025-10-10T01:32:37.4026762Z [?25hDownloading numpy-2.2.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.5 MB) 2025-10-10T01:32:37.5543909Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/16.5 MB ? eta -:--:-- 2025-10-10T01:32:37.5544608Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 111.1 MB/s 0:00:00 2025-10-10T01:32:37.5582379Z [?25hDownloading prometheus_client-0.23.1-py3-none-any.whl (61 kB) 2025-10-10T01:32:37.5645251Z Downloading prometheus_fastapi_instrumentator-7.1.0-py3-none-any.whl (19 kB) 2025-10-10T01:32:37.5705077Z Downloading pydantic_extra_types-2.10.6-py3-none-any.whl (40 kB) 2025-10-10T01:32:37.5769646Z Downloading pycountry-24.6.1-py3-none-any.whl (6.3 MB) 2025-10-10T01:32:37.6334780Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/6.3 MB ? eta -:--:-- 2025-10-10T01:32:37.6335447Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 115.3 MB/s 0:00:00 2025-10-10T01:32:37.6368240Z [?25hDownloading python_multipart-0.0.20-py3-none-any.whl (24 kB) 2025-10-10T01:32:37.6424485Z Downloading pyzmq-27.1.0-cp312-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (840 kB) 2025-10-10T01:32:37.6518966Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/841.0 kB ? eta -:--:-- 2025-10-10T01:32:37.6519707Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 841.0/841.0 kB 87.8 MB/s 0:00:00 2025-10-10T01:32:37.6619465Z [?25hDownloading ray-2.49.2-cp312-cp312-manylinux2014_x86_64.whl (70.1 MB) 2025-10-10T01:32:37.8647338Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/70.1 MB ? eta -:--:-- 2025-10-10T01:32:38.0661382Z  ━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━ 29.4/70.1 MB 146.4 MB/s eta 0:00:01 2025-10-10T01:32:38.2607114Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 60.8/70.1 MB 151.0 MB/s eta 0:00:01 2025-10-10T01:32:38.2608139Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.1/70.1 MB 117.2 MB/s 0:00:00 2025-10-10T01:32:38.2642622Z [?25hDownloading referencing-0.36.2-py3-none-any.whl (26 kB) 2025-10-10T01:32:38.2710372Z Downloading rich_toolkit-0.15.1-py3-none-any.whl (29 kB) 2025-10-10T01:32:38.2780788Z Downloading rich-14.2.0-py3-none-any.whl (243 kB) 2025-10-10T01:32:38.2862370Z Downloading markdown_it_py-4.0.0-py3-none-any.whl (87 kB) 2025-10-10T01:32:38.2938539Z Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB) 2025-10-10T01:32:38.3103906Z Downloading rignore-0.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (951 kB) 2025-10-10T01:32:38.3227072Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/951.1 kB ? eta -:--:-- 2025-10-10T01:32:38.3227731Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 951.1/951.1 kB 70.5 MB/s 0:00:00 2025-10-10T01:32:38.3260591Z [?25hDownloading rpds_py-0.27.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (386 kB) 2025-10-10T01:32:38.3351035Z Downloading sentry_sdk-2.41.0-py2.py3-none-any.whl (370 kB) 2025-10-10T01:32:38.3440719Z Downloading sniffio-1.3.1-py3-none-any.whl (10 kB) 2025-10-10T01:32:38.3521001Z Downloading tiktoken-0.12.0-cp312-cp312-manylinux_2_28_x86_64.whl (1.2 MB) 2025-10-10T01:32:38.3702684Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.2 MB ? eta -:--:-- 2025-10-10T01:32:38.3703317Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 59.5 MB/s 0:00:00 2025-10-10T01:32:38.3743138Z [?25hDownloading regex-2025.9.18-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (802 kB) 2025-10-10T01:32:38.3831934Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/802.0 kB ? eta -:--:-- 2025-10-10T01:32:38.3832555Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 802.0/802.0 kB 84.0 MB/s 0:00:00 2025-10-10T01:32:38.3872052Z [?25hDownloading tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB) 2025-10-10T01:32:38.4087338Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/3.3 MB ? eta -:--:-- 2025-10-10T01:32:38.4087952Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 165.5 MB/s 0:00:00 2025-10-10T01:32:38.4121125Z [?25hDownloading huggingface_hub-0.35.3-py3-none-any.whl (564 kB) 2025-10-10T01:32:38.4203046Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/564.3 kB ? eta -:--:-- 2025-10-10T01:32:38.4203675Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 564.3/564.3 kB 55.5 MB/s 0:00:00 2025-10-10T01:32:38.4247547Z [?25hDownloading hf_xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB) 2025-10-10T01:32:38.4452382Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/3.2 MB ? eta -:--:-- 2025-10-10T01:32:38.4453503Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 162.8 MB/s 0:00:00 2025-10-10T01:32:38.4502436Z [?25hDownloading transformers-4.57.0-py3-none-any.whl (12.0 MB) 2025-10-10T01:32:38.5064764Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/12.0 MB ? eta -:--:-- 2025-10-10T01:32:38.5065413Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 219.0 MB/s 0:00:00 2025-10-10T01:32:38.5101537Z [?25hDownloading safetensors-0.6.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (485 kB) 2025-10-10T01:32:38.5190909Z Downloading typer-0.19.2-py3-none-any.whl (46 kB) 2025-10-10T01:32:38.5249411Z Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB) 2025-10-10T01:32:38.5305777Z Downloading typing_inspection-0.4.2-py3-none-any.whl (14 kB) 2025-10-10T01:32:38.5366953Z Downloading uvicorn-0.37.0-py3-none-any.whl (67 kB) 2025-10-10T01:32:38.5426913Z Downloading httptools-0.6.4-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (510 kB) 2025-10-10T01:32:38.5510384Z Downloading python_dotenv-1.1.1-py3-none-any.whl (20 kB) 2025-10-10T01:32:38.5568460Z Downloading uvloop-0.21.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB) 2025-10-10T01:32:38.5915489Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/4.7 MB ? eta -:--:-- 2025-10-10T01:32:38.5916138Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 137.9 MB/s 0:00:00 2025-10-10T01:32:38.5946496Z [?25hDownloading watchfiles-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (452 kB) 2025-10-10T01:32:38.6033527Z Downloading websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (182 kB) 2025-10-10T01:32:38.6108081Z Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB) 2025-10-10T01:32:38.6201451Z Downloading blake3-1.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (387 kB) 2025-10-10T01:32:38.6312373Z Downloading cbor2-5.7.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (284 kB) 2025-10-10T01:32:38.6395998Z Downloading cloudpickle-3.1.1-py3-none-any.whl (20 kB) 2025-10-10T01:32:38.6663707Z Downloading cupy_cuda12x-13.6.0-cp312-cp312-manylinux2014_x86_64.whl (112.9 MB) 2025-10-10T01:32:38.8692389Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/112.9 MB ? eta -:--:-- 2025-10-10T01:32:39.0705653Z  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.2/112.9 MB 124.8 MB/s eta 0:00:01 2025-10-10T01:32:39.2723292Z  ━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━ 52.4/112.9 MB 130.1 MB/s eta 0:00:01 2025-10-10T01:32:39.4736663Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 76.5/112.9 MB 128.4 MB/s eta 0:00:01 2025-10-10T01:32:39.6754971Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 98.6/112.9 MB 122.9 MB/s eta 0:00:01 2025-10-10T01:32:39.7734083Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 112.7/112.9 MB 126.0 MB/s eta 0:00:01 2025-10-10T01:32:39.7734794Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.9/112.9 MB 102.0 MB/s 0:00:01 2025-10-10T01:32:39.7767665Z [?25hDownloading fastrlock-0.8.3-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl (53 kB) 2025-10-10T01:32:39.7835479Z Downloading einops-0.8.1-py3-none-any.whl (64 kB) 2025-10-10T01:32:39.7905701Z Downloading frozendict-2.4.6-py312-none-any.whl (16 kB) 2025-10-10T01:32:39.7964706Z Downloading msgspec-0.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (213 kB) 2025-10-10T01:32:39.8050377Z Downloading partial_json_parser-0.2.1.1.post6-py3-none-any.whl (10 kB) 2025-10-10T01:32:39.8117635Z Downloading py_cpuinfo-9.0.0-py3-none-any.whl (22 kB) 2025-10-10T01:32:39.8188764Z Downloading pybase64-1.4.2-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl (71 kB) 2025-10-10T01:32:39.8272341Z Downloading python_json_logger-4.0.0-py3-none-any.whl (15 kB) 2025-10-10T01:32:39.8377648Z Downloading sentencepiece-0.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.4 MB) 2025-10-10T01:32:39.8498044Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.4 MB ? eta -:--:-- 2025-10-10T01:32:39.8498679Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 119.8 MB/s 0:00:00 2025-10-10T01:32:39.8527185Z [?25hDownloading setproctitle-1.3.7-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (32 kB) 2025-10-10T01:32:40.4851342Z Installing collected packages: py-cpuinfo, fastrlock, websockets, uvloop, typing_extensions, sniffio, shellingham, setproctitle, sentry-sdk, sentencepiece, safetensors, rpds-py, rignore, regex, pyzmq, python-multipart, python-json-logger, python-dotenv, pycountry, pybase64, prometheus_client, partial-json-parser, outlines_core, numpy, msgspec, mdurl, llvmlite, llguidance, lark, jiter, interegular, httptools, hf-xet, h11, frozendict, einops, dnspython, diskcache, cloudpickle, cbor2, blake3, astor, annotated-types, uvicorn, typing-inspection, tiktoken, referencing, pydantic-core, opencv-python-headless, numba, markdown-it-py, huggingface-hub, httpcore, gguf, email-validator, depyf, cupy-cuda12x, anyio, watchfiles, tokenizers, starlette, rich, pydantic, jsonschema-specifications, httpx, typer, transformers, rich-toolkit, pydantic-extra-types, prometheus-fastapi-instrumentator, openai-harmony, openai, lm-format-enforcer, jsonschema, fastapi, xgrammar, ray, fastapi-cloud-cli, fastapi-cli, compressed-tensors, mistral_common, vllm 2025-10-10T01:32:40.6530001Z [?25l 2025-10-10T01:32:40.6663588Z  ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/82 [uvloop] 2025-10-10T01:32:40.6664103Z  Attempting uninstall: typing_extensions 2025-10-10T01:32:40.6721725Z ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/82 [uvloop] 2025-10-10T01:32:40.6722257Z  Found existing installation: typing_extensions 4.12.2 2025-10-10T01:32:40.6742275Z ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/82 [uvloop] 2025-10-10T01:32:40.6742748Z  Uninstalling typing_extensions-4.12.2: 2025-10-10T01:32:40.8207826Z ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/82 [uvloop] 2025-10-10T01:32:40.9884676Z  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/82 [typing_extensions] 2025-10-10T01:32:41.1374409Z  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/82 [typing_extensions] 2025-10-10T01:32:41.1375330Z  Successfully uninstalled typing_extensions-4.12.2 2025-10-10T01:32:41.1558828Z ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/82 [typing_extensions] 2025-10-10T01:32:41.3237126Z  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/82 [typing_extensions] 2025-10-10T01:32:41.5025074Z  ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  8/82 [sentry-sdk] 2025-10-10T01:32:41.6700720Z  ━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/82 [regex] 2025-10-10T01:32:41.8379860Z  ━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/82 [python-dotenv] 2025-10-10T01:32:41.9617403Z  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/82 [pycountry] 2025-10-10T01:32:41.9617866Z  Attempting uninstall: numpy 2025-10-10T01:32:41.9700450Z ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/82 [pycountry] 2025-10-10T01:32:41.9700978Z  Found existing installation: numpy 1.26.2 2025-10-10T01:32:42.0052503Z ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/82 [pycountry] 2025-10-10T01:32:42.0258227Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:42.0258663Z  Uninstalling numpy-1.26.2: 2025-10-10T01:32:42.1730307Z ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:42.3405790Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:42.3698032Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:42.3698507Z  Successfully uninstalled numpy-1.26.2 2025-10-10T01:32:42.5081253Z ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:42.6756322Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:42.8475251Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.0152312Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.1828274Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.3528013Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.5365107Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.7164142Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.8841171Z  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/82 [numpy] 2025-10-10T01:32:43.9099139Z  ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24/82 [msgspec] 2025-10-10T01:32:43.9099652Z  Attempting uninstall: llvmlite 2025-10-10T01:32:43.9139198Z ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24/82 [msgspec] 2025-10-10T01:32:43.9139697Z  Found existing installation: llvmlite 0.43.0 2025-10-10T01:32:43.9190030Z ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24/82 [msgspec] 2025-10-10T01:32:43.9190469Z  Uninstalling llvmlite-0.43.0: 2025-10-10T01:32:44.0042864Z ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24/82 [msgspec] 2025-10-10T01:32:44.0043369Z  Successfully uninstalled llvmlite-0.43.0 2025-10-10T01:32:44.0515778Z ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24/82 [msgspec] 2025-10-10T01:32:44.2192887Z  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26/82 [llvmlite] 2025-10-10T01:32:44.3869509Z  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26/82 [llvmlite] 2025-10-10T01:32:44.5557650Z  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26/82 [llvmlite] 2025-10-10T01:32:44.7234387Z  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26/82 [llvmlite] 2025-10-10T01:32:44.7730448Z  ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/82 [llguidance] 2025-10-10T01:32:44.7730922Z  Attempting uninstall: lark 2025-10-10T01:32:44.7770299Z ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/82 [llguidance] 2025-10-10T01:32:44.7770806Z  Found existing installation: lark 0.12.0 2025-10-10T01:32:44.7827519Z ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/82 [llguidance] 2025-10-10T01:32:44.7828275Z  Uninstalling lark-0.12.0: 2025-10-10T01:32:44.8070070Z ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/82 [llguidance] 2025-10-10T01:32:44.8070556Z  Successfully uninstalled lark-0.12.0 2025-10-10T01:32:44.8908977Z ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/82 [llguidance] 2025-10-10T01:32:45.0585477Z  ━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━ 30/82 [interegular] 2025-10-10T01:32:45.2261184Z  ━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━ 36/82 [dnspython] 2025-10-10T01:32:45.3952253Z  ━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ 37/82 [diskcache] 2025-10-10T01:32:45.5629192Z  ━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 45/82 [tiktoken] 2025-10-10T01:32:45.7304753Z  ━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━ 48/82 [opencv-python-headless] 2025-10-10T01:32:45.8980133Z  ━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━ 48/82 [opencv-python-headless] 2025-10-10T01:32:46.0559417Z  ━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━ 48/82 [opencv-python-headless] 2025-10-10T01:32:46.0559927Z  Attempting uninstall: numba 2025-10-10T01:32:46.0604962Z ━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━ 48/82 [opencv-python-headless] 2025-10-10T01:32:46.0605481Z  Found existing installation: numba 0.60.0 2025-10-10T01:32:46.0654605Z ━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━ 48/82 [opencv-python-headless] 2025-10-10T01:32:46.1340112Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:46.1340552Z  Uninstalling numba-0.60.0: 2025-10-10T01:32:46.2331457Z ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:46.4007636Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:46.5248639Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:46.5249147Z  Successfully uninstalled numba-0.60.0 2025-10-10T01:32:46.5682795Z ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:46.7422023Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:46.9097911Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:47.0777044Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:47.2453577Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:47.4151949Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:47.5837796Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:47.7569960Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:47.9245605Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:48.0928900Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:48.2640561Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:48.4317023Z  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 49/82 [numba] 2025-10-10T01:32:48.5995126Z  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 50/82 [markdown-it-py] 2025-10-10T01:32:48.7670735Z  ━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━ 51/82 [huggingface-hub] 2025-10-10T01:32:48.9348171Z  ━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━ 53/82 [gguf] 2025-10-10T01:32:49.1022733Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:49.2698067Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:49.4375596Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:49.6051662Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:49.7729022Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:49.9405079Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:50.1080111Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:50.2754643Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:50.4440810Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:50.6121026Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:50.7798891Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 56/82 [cupy-cuda12x] 2025-10-10T01:32:50.9559329Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━ 57/82 [anyio] 2025-10-10T01:32:51.1289184Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━ 61/82 [rich] 2025-10-10T01:32:51.2965933Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 62/82 [pydantic] 2025-10-10T01:32:51.4642759Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 62/82 [pydantic] 2025-10-10T01:32:51.6317948Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:51.7992999Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:51.9668929Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:52.1343905Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:52.3081684Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:52.4813131Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:52.6507387Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:52.8183257Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:52.9891221Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:53.1570645Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:53.3274650Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:53.4974182Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:53.6649914Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:53.8324979Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:54.0024464Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:54.1721353Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:54.3451134Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:54.5177318Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:54.6903517Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:54.8640284Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:55.0314782Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:55.1998033Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:55.3681697Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:55.5372403Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:55.7059995Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:55.8736093Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:56.0414553Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:56.2091121Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:56.3772829Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:56.5448500Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:56.7239380Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:56.8918197Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:57.0597933Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 66/82 [transformers] 2025-10-10T01:32:57.2277927Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 67/82 [rich-toolkit] 2025-10-10T01:32:57.3954172Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━ 69/82 [prometheus-fastapi-instrumentator] 2025-10-10T01:32:57.5646214Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 71/82 [openai] 2025-10-10T01:32:57.7323437Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 71/82 [openai] 2025-10-10T01:32:57.8998250Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 71/82 [openai] 2025-10-10T01:32:58.0678508Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 71/82 [openai] 2025-10-10T01:32:58.2358030Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 71/82 [openai] 2025-10-10T01:32:58.4038847Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━ 73/82 [jsonschema] 2025-10-10T01:32:58.5716628Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 74/82 [fastapi] 2025-10-10T01:32:58.7392498Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 75/82 [xgrammar] 2025-10-10T01:32:58.9066579Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:32:59.0742164Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:32:59.2417141Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:32:59.4093768Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:32:59.5793961Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:32:59.7477248Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:32:59.9157912Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:00.0851879Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:00.2548077Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:00.4227856Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:00.5906677Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:00.7582681Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:00.9260195Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:01.0954661Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:01.2629253Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:01.4304599Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:01.5979868Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:01.7656802Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:01.9332422Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:02.1014266Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:02.2722660Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:02.4403150Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 76/82 [ray] 2025-10-10T01:33:02.6080533Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 77/82 [fastapi-cloud-cli] 2025-10-10T01:33:02.7757118Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━ 79/82 [compressed-tensors] 2025-10-10T01:33:02.9433518Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 80/82 [mistral_common] 2025-10-10T01:33:03.1110039Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:03.2785463Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:03.4466790Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:03.6143536Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:03.7820492Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:03.9496858Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:04.1172904Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:04.2850205Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:04.4525145Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:04.6202497Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:04.7878038Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:04.9555491Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:05.1231879Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:05.2907287Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:05.4584345Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:05.6261198Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:05.7937404Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:05.9614544Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:06.1291196Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:06.2968263Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:06.4644843Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:06.6321493Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:06.7997856Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:06.9674980Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:07.1351288Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:07.3027295Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:07.4704118Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:07.6378888Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:07.8059257Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:07.9747933Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:08.1423927Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:08.3102257Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:08.4784688Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:08.6478034Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:08.8157415Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:08.9847620Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:09.1528753Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:09.2653275Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 81/82 [vllm] 2025-10-10T01:33:09.2653812Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82/82 [vllm] 2025-10-10T01:33:09.2654148Z [?25h 2025-10-10T01:33:09.2798989Z Successfully installed annotated-types-0.7.0 anyio-4.11.0 astor-0.8.1 blake3-1.0.7 cbor2-5.7.0 cloudpickle-3.1.1 compressed-tensors-0.11.0 cupy-cuda12x-13.6.0 depyf-0.19.0 diskcache-5.6.3 dnspython-2.8.0 einops-0.8.1 email-validator-2.3.0 fastapi-0.118.2 fastapi-cli-0.0.13 fastapi-cloud-cli-0.3.1 fastrlock-0.8.3 frozendict-2.4.6 gguf-0.17.1 h11-0.16.0 hf-xet-1.1.10 httpcore-1.0.9 httptools-0.6.4 httpx-0.28.1 huggingface-hub-0.35.3 interegular-0.3.3 jiter-0.11.0 jsonschema-4.25.1 jsonschema-specifications-2025.9.1 lark-1.2.2 llguidance-0.7.30 llvmlite-0.44.0 lm-format-enforcer-0.11.3 markdown-it-py-4.0.0 mdurl-0.1.2 mistral_common-1.8.5 msgspec-0.19.0 numba-0.61.2 numpy-2.2.6 openai-2.3.0 openai-harmony-0.0.4 opencv-python-headless-4.12.0.88 outlines_core-0.2.11 partial-json-parser-0.2.1.1.post6 prometheus-fastapi-instrumentator-7.1.0 prometheus_client-0.23.1 py-cpuinfo-9.0.0 pybase64-1.4.2 pycountry-24.6.1 pydantic-2.12.0 pydantic-core-2.41.1 pydantic-extra-types-2.10.6 python-dotenv-1.1.1 python-json-logger-4.0.0 python-multipart-0.0.20 pyzmq-27.1.0 ray-2.49.2 referencing-0.36.2 regex-2025.9.18 rich-14.2.0 rich-toolkit-0.15.1 rignore-0.7.0 rpds-py-0.27.1 safetensors-0.6.2 sentencepiece-0.2.1 sentry-sdk-2.41.0 setproctitle-1.3.7 shellingham-1.5.4 sniffio-1.3.1 starlette-0.48.0 tiktoken-0.12.0 tokenizers-0.22.1 transformers-4.57.0 typer-0.19.2 typing-inspection-0.4.2 typing_extensions-4.15.0 uvicorn-0.37.0 uvloop-0.21.0 vllm-0.11.0rc2.dev157+g0ad9951c4.d20251010 watchfiles-1.1.0 websockets-15.0.1 xgrammar-0.1.25 2025-10-10T01:33:10.4004670Z 2025-10-10 01:33:10,399 [INFO] cli.lib.common.pip_helper: Installing /var/lib/jenkins/workspace/dist/external/vllm/wheels/flashinfer-python/flashinfer_python-0.2.14.post1-cp39-abi3-linux_x86_64.whl... 2025-10-10T01:33:10.4006526Z 2025-10-10 01:33:10,400 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/external/vllm/wheels/flashinfer-python/flashinfer_python-0.2.14.post1-cp39-abi3-linux_x86_64.whl 2025-10-10T01:33:10.4008100Z 2025-10-10 01:33:10,400 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m pip install /var/lib/jenkins/workspace/dist/external/vllm/wheels/flashinfer-python/flashinfer_python-0.2.14.post1-cp39-abi3-linux_x86_64.whl 2025-10-10T01:33:10.6979579Z Processing /var/lib/jenkins/workspace/dist/external/vllm/wheels/flashinfer-python/flashinfer_python-0.2.14.post1-cp39-abi3-linux_x86_64.whl 2025-10-10T01:33:10.7723412Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from flashinfer-python==0.2.14.post1) (2.2.6) 2025-10-10T01:33:10.7726062Z Requirement already satisfied: torch in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from flashinfer-python==0.2.14.post1) (2.10.0a0+git344e636) 2025-10-10T01:33:10.7728246Z Requirement already satisfied: ninja in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from flashinfer-python==0.2.14.post1) (1.10.2) 2025-10-10T01:33:10.7730719Z Requirement already satisfied: requests in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from flashinfer-python==0.2.14.post1) (2.32.5) 2025-10-10T01:33:10.8199510Z Collecting cuda-python<=12.9 (from flashinfer-python==0.2.14.post1) 2025-10-10T01:33:10.8488625Z Downloading cuda_python-12.9.0-py3-none-any.whl.metadata (4.6 kB) 2025-10-10T01:33:10.8694513Z Collecting pynvml (from flashinfer-python==0.2.14.post1) 2025-10-10T01:33:10.8746138Z Downloading pynvml-13.0.1-py3-none-any.whl.metadata (5.6 kB) 2025-10-10T01:33:10.8819358Z Requirement already satisfied: einops in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from flashinfer-python==0.2.14.post1) (0.8.1) 2025-10-10T01:33:10.8940692Z Collecting packaging>=24.2 (from flashinfer-python==0.2.14.post1) 2025-10-10T01:33:10.8973712Z Downloading packaging-25.0-py3-none-any.whl.metadata (3.3 kB) 2025-10-10T01:33:10.9211381Z Collecting nvidia-cudnn-frontend>=1.13.0 (from flashinfer-python==0.2.14.post1) 2025-10-10T01:33:10.9267743Z Downloading nvidia_cudnn_frontend-1.14.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.7 kB) 2025-10-10T01:33:10.9340552Z Requirement already satisfied: cuda-bindings~=12.9.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from cuda-python<=12.9->flashinfer-python==0.2.14.post1) (12.9.2) 2025-10-10T01:33:10.9348430Z Requirement already satisfied: cuda-pathfinder~=1.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from cuda-bindings~=12.9.0->cuda-python<=12.9->flashinfer-python==0.2.14.post1) (1.3.0) 2025-10-10T01:33:10.9382444Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (3.18.0) 2025-10-10T01:33:10.9386121Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (4.15.0) 2025-10-10T01:33:10.9390845Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (78.1.1) 2025-10-10T01:33:10.9394409Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (1.13.3) 2025-10-10T01:33:10.9398257Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (2.8.8) 2025-10-10T01:33:10.9401842Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (3.1.6) 2025-10-10T01:33:10.9405561Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch->flashinfer-python==0.2.14.post1) (2025.9.0) 2025-10-10T01:33:10.9515205Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch->flashinfer-python==0.2.14.post1) (1.3.0) 2025-10-10T01:33:10.9563529Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2->torch->flashinfer-python==0.2.14.post1) (3.0.3) 2025-10-10T01:33:10.9749115Z Collecting nvidia-ml-py>=12.0.0 (from pynvml->flashinfer-python==0.2.14.post1) 2025-10-10T01:33:10.9799478Z Downloading nvidia_ml_py-13.580.82-py3-none-any.whl.metadata (9.6 kB) 2025-10-10T01:33:10.9884585Z Requirement already satisfied: charset_normalizer<4,>=2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests->flashinfer-python==0.2.14.post1) (3.4.3) 2025-10-10T01:33:10.9888936Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests->flashinfer-python==0.2.14.post1) (3.10) 2025-10-10T01:33:10.9893046Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests->flashinfer-python==0.2.14.post1) (2.5.0) 2025-10-10T01:33:10.9896996Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from requests->flashinfer-python==0.2.14.post1) (2025.10.5) 2025-10-10T01:33:11.0081930Z Downloading cuda_python-12.9.0-py3-none-any.whl (7.5 kB) 2025-10-10T01:33:11.0267711Z Downloading nvidia_cudnn_frontend-1.14.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.8 MB) 2025-10-10T01:33:11.0621966Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.8 MB ? eta -:--:-- 2025-10-10T01:33:11.0622648Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 81.5 MB/s 0:00:00 2025-10-10T01:33:11.0654965Z [?25hDownloading packaging-25.0-py3-none-any.whl (66 kB) 2025-10-10T01:33:11.0768516Z Downloading pynvml-13.0.1-py3-none-any.whl (28 kB) 2025-10-10T01:33:11.0909957Z Downloading nvidia_ml_py-13.580.82-py3-none-any.whl (49 kB) 2025-10-10T01:33:11.3854951Z Installing collected packages: nvidia-ml-py, pynvml, packaging, nvidia-cudnn-frontend, cuda-python, flashinfer-python 2025-10-10T01:33:11.4578955Z [?25l 2025-10-10T01:33:11.4579297Z  Attempting uninstall: packaging 2025-10-10T01:33:11.4623404Z 2025-10-10T01:33:11.4623703Z  Found existing installation: packaging 23.1 2025-10-10T01:33:11.4654266Z 2025-10-10T01:33:11.4654500Z  Uninstalling packaging-23.1: 2025-10-10T01:33:11.4755686Z 2025-10-10T01:33:11.4755961Z  Successfully uninstalled packaging-23.1 2025-10-10T01:33:11.5536336Z 2025-10-10T01:33:11.7216406Z  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 3/6 [nvidia-cudnn-frontend] 2025-10-10T01:33:11.8891330Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:12.0568831Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:12.2243709Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:12.3920813Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:12.5597404Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:12.7273683Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:12.8949440Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:13.0625401Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:13.1739081Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 5/6 [flashinfer-python] 2025-10-10T01:33:13.1739696Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6/6 [flashinfer-python] 2025-10-10T01:33:13.1740073Z [?25h 2025-10-10T01:33:13.1880066Z Successfully installed cuda-python-12.9.0 flashinfer-python-0.2.14.post1 nvidia-cudnn-frontend-1.14.1 nvidia-ml-py-13.580.82 packaging-25.0 pynvml-13.0.1 2025-10-10T01:33:13.2472756Z 2025-10-10 01:33:13,246 [INFO] cli.lib.core.vllm.vllm_test: Done. Installed vllm wheels 2025-10-10T01:33:13.2473908Z 2025-10-10 01:33:13,247 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m uv pip install -e tests/vllm_test_utils 2025-10-10T01:33:13.2474842Z 2025-10-10 01:33:13,247 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m uv pip install -e tests/vllm_test_utils 2025-10-10T01:33:13.3590433Z Using Python 3.12.11 environment at: /opt/conda/envs/py_3.12 2025-10-10T01:33:14.0531810Z Resolved 1 package in 648ms 2025-10-10T01:33:14.0552991Z Building vllm-test-utils @ file:///var/lib/jenkins/workspace/vllm/tests/vllm_test_utils 2025-10-10T01:33:14.2329428Z Built vllm-test-utils @ file:///var/lib/jenkins/workspace/vllm/tests/vllm_test_utils 2025-10-10T01:33:14.2342124Z Prepared 1 package in 180ms 2025-10-10T01:33:14.2361839Z Installed 1 package in 1ms 2025-10-10T01:33:14.2363375Z + vllm-test-utils==0.1 (from file:///var/lib/jenkins/workspace/vllm/tests/vllm_test_utils) 2025-10-10T01:33:14.2546925Z 2025-10-10 01:33:14,254 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m uv pip install hf_transfer 2025-10-10T01:33:14.2547818Z 2025-10-10 01:33:14,254 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m uv pip install hf_transfer 2025-10-10T01:33:14.3083461Z Using Python 3.12.11 environment at: /opt/conda/envs/py_3.12 2025-10-10T01:33:14.3433224Z Resolved 1 package in 29ms 2025-10-10T01:33:14.3465618Z Downloading hf-transfer (3.4MiB) 2025-10-10T01:33:14.3761046Z Downloading hf-transfer 2025-10-10T01:33:14.3764079Z Prepared 1 package in 33ms 2025-10-10T01:33:14.3809399Z Installed 1 package in 4ms 2025-10-10T01:33:14.3809980Z + hf-transfer==0.1.9 2025-10-10T01:33:14.3841263Z 2025-10-10 01:33:14,383 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python use_existing_torch.py 2025-10-10T01:33:14.4204351Z >>> cleaning requirements/build.txt 2025-10-10T01:33:14.4204790Z removed: 2025-10-10T01:33:14.4206004Z <<< done cleaning requirements/build.txt 2025-10-10T01:33:14.4206255Z 2025-10-10T01:33:14.4206442Z >>> cleaning requirements/cpu-build.txt 2025-10-10T01:33:14.4206903Z removed: 2025-10-10T01:33:14.4207694Z <<< done cleaning requirements/cpu-build.txt 2025-10-10T01:33:14.4207922Z 2025-10-10T01:33:14.4208048Z >>> cleaning requirements/dev.txt 2025-10-10T01:33:14.4208469Z <<< done cleaning requirements/dev.txt 2025-10-10T01:33:14.4208713Z 2025-10-10T01:33:14.4208796Z >>> cleaning requirements/docs.txt 2025-10-10T01:33:14.4208997Z removed: 2025-10-10T01:33:14.4210077Z <<< done cleaning requirements/docs.txt 2025-10-10T01:33:14.4210380Z 2025-10-10T01:33:14.4210555Z >>> cleaning requirements/lint.txt 2025-10-10T01:33:14.4210863Z <<< done cleaning requirements/lint.txt 2025-10-10T01:33:14.4211051Z 2025-10-10T01:33:14.4211162Z >>> cleaning requirements/rocm-build.txt 2025-10-10T01:33:14.4211572Z removed: 2025-10-10T01:33:14.4212293Z <<< done cleaning requirements/rocm-build.txt 2025-10-10T01:33:14.4212523Z 2025-10-10T01:33:14.4212707Z >>> cleaning requirements/rocm-test.txt 2025-10-10T01:33:14.4213081Z <<< done cleaning requirements/rocm-test.txt 2025-10-10T01:33:14.4213306Z 2025-10-10T01:33:14.4213413Z >>> cleaning requirements/tpu.txt 2025-10-10T01:33:14.4213795Z removed: 2025-10-10T01:33:14.4214563Z <<< done cleaning requirements/tpu.txt 2025-10-10T01:33:14.4214768Z 2025-10-10T01:33:14.4214865Z >>> cleaning requirements/common.txt 2025-10-10T01:33:14.4215286Z <<< done cleaning requirements/common.txt 2025-10-10T01:33:14.4215477Z 2025-10-10T01:33:14.4215575Z >>> cleaning requirements/cpu.txt 2025-10-10T01:33:14.4215815Z removed: 2025-10-10T01:33:14.4216889Z <<< done cleaning requirements/cpu.txt 2025-10-10T01:33:14.4217090Z 2025-10-10T01:33:14.4217183Z >>> cleaning requirements/cuda.txt 2025-10-10T01:33:14.4217429Z removed: 2025-10-10T01:33:14.4218531Z <<< done cleaning requirements/cuda.txt 2025-10-10T01:33:14.4218687Z 2025-10-10T01:33:14.4218779Z >>> cleaning requirements/kv_connectors.txt 2025-10-10T01:33:14.4219254Z <<< done cleaning requirements/kv_connectors.txt 2025-10-10T01:33:14.4219519Z 2025-10-10T01:33:14.4219646Z >>> cleaning requirements/nightly_torch_test.txt 2025-10-10T01:33:14.4220295Z <<< done cleaning requirements/nightly_torch_test.txt 2025-10-10T01:33:14.4220489Z 2025-10-10T01:33:14.4220580Z >>> cleaning requirements/rocm.txt 2025-10-10T01:33:14.4220827Z <<< done cleaning requirements/rocm.txt 2025-10-10T01:33:14.4220987Z 2025-10-10T01:33:14.4221065Z >>> cleaning requirements/test.txt 2025-10-10T01:33:14.4221983Z removed: 2025-10-10T01:33:14.4225574Z <<< done cleaning requirements/test.txt 2025-10-10T01:33:14.4225825Z 2025-10-10T01:33:14.4225911Z >>> cleaning requirements/xpu.txt 2025-10-10T01:33:14.4226264Z removed: 2025-10-10T01:33:14.4227272Z <<< done cleaning requirements/xpu.txt 2025-10-10T01:33:14.4227458Z 2025-10-10T01:33:14.4227542Z >>> cleaning pyproject.toml 2025-10-10T01:33:14.4228231Z removed: 2025-10-10T01:33:14.4229809Z <<< done cleaning pyproject.toml 2025-10-10T01:33:14.4229968Z 2025-10-10T01:33:14.4284930Z 2025-10-10 01:33:14,428 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m uv pip install -r requirements/common.txt 2025-10-10T01:33:14.4286404Z 2025-10-10 01:33:14,428 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m uv pip install -r requirements/common.txt 2025-10-10T01:33:14.4770592Z Using Python 3.12.11 environment at: /opt/conda/envs/py_3.12 2025-10-10T01:33:14.4927222Z Audited 50 packages in 17ms 2025-10-10T01:33:14.4946466Z 2025-10-10 01:33:14,494 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m uv pip install -r requirements/build.txt 2025-10-10T01:33:14.4947476Z 2025-10-10 01:33:14,494 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m uv pip install -r requirements/build.txt 2025-10-10T01:33:14.5415634Z Using Python 3.12.11 environment at: /opt/conda/envs/py_3.12 2025-10-10T01:33:14.6138569Z Resolved 11 packages in 67ms 2025-10-10T01:33:14.6280269Z Prepared 1 package in 14ms 2025-10-10T01:33:14.6344624Z Installed 1 package in 6ms 2025-10-10T01:33:14.6345120Z + setuptools-scm==9.2.0 2025-10-10T01:33:14.6397694Z 2025-10-10 01:33:14,639 [INFO] cli.lib.core.vllm.vllm_test: generate test.txt from requirements/test.in with local torch whls 2025-10-10T01:33:15.0046768Z 2025-10-10 01:33:15,004 [INFO] cli.lib.core.vllm.vllm_test: [INFO] Updated requirements/test.in 2025-10-10T01:33:15.0050856Z 2025-10-10 01:33:15,004 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m uv pip compile requirements/test.in -o test.txt --index-strategy unsafe-best-match --constraint snapshot_constraint.txt --torch-backend cu128 2025-10-10T01:33:18.1815890Z Resolved 324 packages in 3.12s 2025-10-10T01:33:18.1816377Z # This file was autogenerated by uv via the following command: 2025-10-10T01:33:18.1817137Z # uv pip compile requirements/test.in -o test.txt --index-strategy unsafe-best-match --constraint snapshot_constraint.txt --torch-backend cu128 2025-10-10T01:33:18.1866382Z absl-py==2.1.0 2025-10-10T01:33:18.1866811Z  # via 2025-10-10T01:33:18.1867195Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1867547Z # rouge-score 2025-10-10T01:33:18.1867883Z accelerate==1.0.1 2025-10-10T01:33:18.1868317Z  # via 2025-10-10T01:33:18.1868589Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1869049Z # lm-eval 2025-10-10T01:33:18.1869373Z # peft 2025-10-10T01:33:18.1869724Z aenum==3.1.16 2025-10-10T01:33:18.1870106Z  # via 2025-10-10T01:33:18.1870484Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1871220Z # lightly 2025-10-10T01:33:18.1871703Z affine==2.4.0 2025-10-10T01:33:18.1872171Z  # via 2025-10-10T01:33:18.1872495Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1872956Z # rasterio 2025-10-10T01:33:18.1873365Z aiohappyeyeballs==2.4.3 2025-10-10T01:33:18.1873771Z  # via 2025-10-10T01:33:18.1874463Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1874943Z # aiohttp 2025-10-10T01:33:18.1875273Z aiohttp==3.10.11 2025-10-10T01:33:18.1875629Z  # via 2025-10-10T01:33:18.1875930Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1876399Z # aiohttp-cors 2025-10-10T01:33:18.1876789Z # datasets 2025-10-10T01:33:18.1877054Z # fsspec 2025-10-10T01:33:18.1877287Z # lm-eval 2025-10-10T01:33:18.1877576Z # ray 2025-10-10T01:33:18.1877779Z aiohttp-cors==0.8.1 2025-10-10T01:33:18.1878081Z  # via 2025-10-10T01:33:18.1878360Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1878634Z # ray 2025-10-10T01:33:18.1878931Z aiosignal==1.3.1 2025-10-10T01:33:18.1879319Z  # via 2025-10-10T01:33:18.1879552Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1879944Z # aiohttp 2025-10-10T01:33:18.1880220Z albucore==0.0.16 2025-10-10T01:33:18.1880548Z  # via 2025-10-10T01:33:18.1880837Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1881247Z # terratorch 2025-10-10T01:33:18.1881467Z albumentations==1.4.6 2025-10-10T01:33:18.1882037Z  # via 2025-10-10T01:33:18.1882366Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1882963Z # terratorch 2025-10-10T01:33:18.1883248Z alembic==1.16.4 2025-10-10T01:33:18.1883488Z  # via 2025-10-10T01:33:18.1883791Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1884224Z # mlflow 2025-10-10T01:33:18.1884533Z annotated-types==0.7.0 2025-10-10T01:33:18.1884868Z  # via 2025-10-10T01:33:18.1885137Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1885551Z # pydantic 2025-10-10T01:33:18.1885829Z antlr4-python3-runtime==4.9.3 2025-10-10T01:33:18.1886195Z  # via 2025-10-10T01:33:18.1886528Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1886752Z # hydra-core 2025-10-10T01:33:18.1887066Z # omegaconf 2025-10-10T01:33:18.1887320Z anyio==4.6.2.post1 2025-10-10T01:33:18.1887637Z  # via 2025-10-10T01:33:18.1887852Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1888172Z # httpx 2025-10-10T01:33:18.1888515Z # starlette 2025-10-10T01:33:18.1888796Z argcomplete==3.5.1 2025-10-10T01:33:18.1889133Z  # via 2025-10-10T01:33:18.1889404Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1889877Z # datamodel-code-generator 2025-10-10T01:33:18.1890244Z arrow==1.3.0 2025-10-10T01:33:18.1890525Z  # via 2025-10-10T01:33:18.1890812Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1891238Z # isoduration 2025-10-10T01:33:18.1891502Z attrs==24.2.0 2025-10-10T01:33:18.1891785Z  # via 2025-10-10T01:33:18.1891988Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1892291Z # aiohttp 2025-10-10T01:33:18.1892478Z # fiona 2025-10-10T01:33:18.1892706Z # hypothesis 2025-10-10T01:33:18.1892866Z # jsonlines 2025-10-10T01:33:18.1893027Z # jsonschema 2025-10-10T01:33:18.1893207Z # pytest-subtests 2025-10-10T01:33:18.1893489Z # rasterio 2025-10-10T01:33:18.1893700Z # referencing 2025-10-10T01:33:18.1893968Z audioread==3.0.1 2025-10-10T01:33:18.1894195Z  # via 2025-10-10T01:33:18.1894409Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1894826Z # librosa 2025-10-10T01:33:18.1895131Z backoff==2.2.1 2025-10-10T01:33:18.1895399Z  # via 2025-10-10T01:33:18.1895569Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1895779Z # -r requirements/test.in 2025-10-10T01:33:18.1896015Z # schemathesis 2025-10-10T01:33:18.1896454Z bitsandbytes==0.46.1 2025-10-10T01:33:18.1896656Z  # via 2025-10-10T01:33:18.1896831Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1897043Z # -r requirements/test.in 2025-10-10T01:33:18.1897267Z # lightning 2025-10-10T01:33:18.1897437Z black==24.10.0 2025-10-10T01:33:18.1897612Z  # via 2025-10-10T01:33:18.1897784Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1898365Z # datamodel-code-generator 2025-10-10T01:33:18.1898579Z blinker==1.9.0 2025-10-10T01:33:18.1898850Z  # via 2025-10-10T01:33:18.1899076Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1899460Z # flask 2025-10-10T01:33:18.1899737Z blobfile==3.0.0 2025-10-10T01:33:18.1900071Z  # via 2025-10-10T01:33:18.1900365Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1900793Z # -r requirements/test.in 2025-10-10T01:33:18.1901005Z bm25s==0.2.13 2025-10-10T01:33:18.1901181Z  # via 2025-10-10T01:33:18.1901384Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1901810Z # mteb 2025-10-10T01:33:18.1902092Z boto3==1.35.57 2025-10-10T01:33:18.1902372Z  # via 2025-10-10T01:33:18.1902564Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1902954Z # runai-model-streamer-s3 2025-10-10T01:33:18.1903366Z # tensorizer 2025-10-10T01:33:18.1903679Z botocore==1.35.57 2025-10-10T01:33:18.1904025Z  # via 2025-10-10T01:33:18.1904319Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1904649Z # boto3 2025-10-10T01:33:18.1905185Z # s3transfer 2025-10-10T01:33:18.1905547Z bounded-pool-executor==0.0.3 2025-10-10T01:33:18.1905863Z  # via 2025-10-10T01:33:18.1906140Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1906552Z # pqdm 2025-10-10T01:33:18.1906868Z buildkite-test-collector==0.1.9 2025-10-10T01:33:18.1907251Z  # via 2025-10-10T01:33:18.1907542Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1907974Z # -r requirements/test.in 2025-10-10T01:33:18.1908345Z cachetools==5.5.2 2025-10-10T01:33:18.1908666Z  # via 2025-10-10T01:33:18.1908967Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1909319Z # google-auth 2025-10-10T01:33:18.1909643Z # mlflow-skinny 2025-10-10T01:33:18.1909968Z certifi==2024.8.30 2025-10-10T01:33:18.1910303Z  # via 2025-10-10T01:33:18.1910599Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1910938Z # fiona 2025-10-10T01:33:18.1911190Z # httpcore 2025-10-10T01:33:18.1911469Z # httpx 2025-10-10T01:33:18.1911718Z # lightly 2025-10-10T01:33:18.1911913Z # pyogrio 2025-10-10T01:33:18.1912160Z # pyproj 2025-10-10T01:33:18.1912389Z # rasterio 2025-10-10T01:33:18.1912733Z # requests 2025-10-10T01:33:18.1913033Z cffi==1.17.1 2025-10-10T01:33:18.1913347Z  # via 2025-10-10T01:33:18.1913636Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1914043Z # soundfile 2025-10-10T01:33:18.1914336Z chardet==5.2.0 2025-10-10T01:33:18.1914603Z  # via 2025-10-10T01:33:18.1914906Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1915296Z # mbstrdecoder 2025-10-10T01:33:18.1915634Z charset-normalizer==3.4.0 2025-10-10T01:33:18.1915997Z  # via 2025-10-10T01:33:18.1916265Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1916684Z # requests 2025-10-10T01:33:18.1917007Z click==8.1.7 2025-10-10T01:33:18.1917311Z  # via 2025-10-10T01:33:18.1917595Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1917965Z # black 2025-10-10T01:33:18.1918346Z # click-plugins 2025-10-10T01:33:18.1918617Z # cligj 2025-10-10T01:33:18.1918770Z # fiona 2025-10-10T01:33:18.1918919Z # flask 2025-10-10T01:33:18.1919195Z # jiwer 2025-10-10T01:33:18.1919462Z # mlflow-skinny 2025-10-10T01:33:18.1919761Z # nltk 2025-10-10T01:33:18.1920004Z # rasterio 2025-10-10T01:33:18.1920278Z # ray 2025-10-10T01:33:18.1920537Z # schemathesis 2025-10-10T01:33:18.1920842Z # typer 2025-10-10T01:33:18.1921145Z # uvicorn 2025-10-10T01:33:18.1921445Z click-plugins==1.1.1.2 2025-10-10T01:33:18.1921817Z  # via 2025-10-10T01:33:18.1922018Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1922357Z # fiona 2025-10-10T01:33:18.1922706Z # rasterio 2025-10-10T01:33:18.1923002Z cligj==0.7.2 2025-10-10T01:33:18.1923561Z  # via 2025-10-10T01:33:18.1923878Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1924119Z # fiona 2025-10-10T01:33:18.1924474Z # rasterio 2025-10-10T01:33:18.1924691Z cloudpickle==3.1.1 2025-10-10T01:33:18.1924885Z  # via 2025-10-10T01:33:18.1939138Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1939530Z # mlflow-skinny 2025-10-10T01:33:18.1939739Z colorama==0.4.6 2025-10-10T01:33:18.1939940Z  # via 2025-10-10T01:33:18.1940124Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1940335Z # sacrebleu 2025-10-10T01:33:18.1940506Z # schemathesis 2025-10-10T01:33:18.1940730Z # tqdm-multiprocess 2025-10-10T01:33:18.1940934Z colorful==0.5.6 2025-10-10T01:33:18.1941117Z  # via 2025-10-10T01:33:18.1941287Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1941515Z # ray 2025-10-10T01:33:18.1941804Z contourpy==1.3.0 2025-10-10T01:33:18.1941992Z  # via 2025-10-10T01:33:18.1942190Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1942432Z # matplotlib 2025-10-10T01:33:18.1942724Z coverage==7.10.6 2025-10-10T01:33:18.1942909Z  # via 2025-10-10T01:33:18.1943078Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1943328Z # pytest-cov 2025-10-10T01:33:18.1943514Z cramjam==2.9.0 2025-10-10T01:33:18.1943689Z  # via 2025-10-10T01:33:18.1943863Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1944108Z # fastparquet 2025-10-10T01:33:18.1944297Z cupy-cuda12x==13.6.0 2025-10-10T01:33:18.1944496Z  # via 2025-10-10T01:33:18.1944667Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1945045Z # ray 2025-10-10T01:33:18.1945206Z cycler==0.12.1 2025-10-10T01:33:18.1945375Z  # via 2025-10-10T01:33:18.1945537Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1945764Z # matplotlib 2025-10-10T01:33:18.1945945Z databricks-sdk==0.59.0 2025-10-10T01:33:18.1946131Z  # via 2025-10-10T01:33:18.1946301Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1946543Z # mlflow-skinny 2025-10-10T01:33:18.1946752Z datamodel-code-generator==0.26.3 2025-10-10T01:33:18.1946972Z  # via 2025-10-10T01:33:18.1947141Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1947391Z # -r requirements/test.in 2025-10-10T01:33:18.1947602Z dataproperty==1.0.1 2025-10-10T01:33:18.1947785Z  # via 2025-10-10T01:33:18.1947951Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1948155Z # pytablewriter 2025-10-10T01:33:18.1948362Z # tabledata 2025-10-10T01:33:18.1948529Z datasets==3.0.2 2025-10-10T01:33:18.1948714Z  # via 2025-10-10T01:33:18.1948882Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1949069Z # evaluate 2025-10-10T01:33:18.1949226Z # lm-eval 2025-10-10T01:33:18.1949405Z # mteb 2025-10-10T01:33:18.1949568Z decorator==5.1.1 2025-10-10T01:33:18.1949743Z  # via 2025-10-10T01:33:18.1949925Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1950153Z # librosa 2025-10-10T01:33:18.1950319Z decord==0.6.0 2025-10-10T01:33:18.1950490Z  # via 2025-10-10T01:33:18.1950654Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1950897Z # -r requirements/test.in 2025-10-10T01:33:18.1951100Z dill==0.3.8 2025-10-10T01:33:18.1951264Z  # via 2025-10-10T01:33:18.1951429Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1951624Z # datasets 2025-10-10T01:33:18.1951777Z # evaluate 2025-10-10T01:33:18.1951919Z # lm-eval 2025-10-10T01:33:18.1952108Z # multiprocess 2025-10-10T01:33:18.1952292Z distlib==0.3.9 2025-10-10T01:33:18.1952464Z  # via 2025-10-10T01:33:18.1952635Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1952863Z # virtualenv 2025-10-10T01:33:18.1953046Z dnspython==2.7.0 2025-10-10T01:33:18.1953223Z  # via 2025-10-10T01:33:18.1953378Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1953704Z # email-validator 2025-10-10T01:33:18.1953897Z docker==7.1.0 2025-10-10T01:33:18.1954063Z  # via 2025-10-10T01:33:18.1954230Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1954448Z # mlflow 2025-10-10T01:33:18.1954609Z docopt==0.6.2 2025-10-10T01:33:18.1954772Z  # via 2025-10-10T01:33:18.1955029Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1955263Z # num2words 2025-10-10T01:33:18.1955454Z docstring-parser==0.17.0 2025-10-10T01:33:18.1955654Z  # via 2025-10-10T01:33:18.1955817Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1956048Z # jsonargparse 2025-10-10T01:33:18.1956247Z efficientnet-pytorch==0.7.1 2025-10-10T01:33:18.1956514Z  # via segmentation-models-pytorch 2025-10-10T01:33:18.1956753Z einops==0.8.1 2025-10-10T01:33:18.1956932Z  # via 2025-10-10T01:33:18.1957106Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1957314Z # -r requirements/test.in 2025-10-10T01:33:18.1957530Z # encodec 2025-10-10T01:33:18.1957687Z # terratorch 2025-10-10T01:33:18.1957853Z # torchgeo 2025-10-10T01:33:18.1958125Z # vector-quantize-pytorch 2025-10-10T01:33:18.1958360Z # vocos 2025-10-10T01:33:18.1958523Z einx==0.3.0 2025-10-10T01:33:18.1958694Z  # via 2025-10-10T01:33:18.1958852Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1959266Z # vector-quantize-pytorch 2025-10-10T01:33:18.1959488Z email-validator==2.2.0 2025-10-10T01:33:18.1959687Z  # via 2025-10-10T01:33:18.1959846Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1960074Z # pydantic 2025-10-10T01:33:18.1960247Z encodec==0.1.1 2025-10-10T01:33:18.1960414Z  # via 2025-10-10T01:33:18.1960571Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1960783Z # vocos 2025-10-10T01:33:18.1960944Z eval-type-backport==0.2.2 2025-10-10T01:33:18.1961142Z  # via 2025-10-10T01:33:18.1961298Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1961517Z # mteb 2025-10-10T01:33:18.1961670Z evaluate==0.4.3 2025-10-10T01:33:18.1961847Z  # via 2025-10-10T01:33:18.1962010Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1962233Z # lm-eval 2025-10-10T01:33:18.1962404Z fastapi==0.116.1 2025-10-10T01:33:18.1962582Z  # via 2025-10-10T01:33:18.1962744Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1962974Z # mlflow-skinny 2025-10-10T01:33:18.1963160Z fastparquet==2024.11.0 2025-10-10T01:33:18.1963352Z  # via 2025-10-10T01:33:18.1963511Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1963741Z # genai-perf 2025-10-10T01:33:18.1963915Z fastrlock==0.8.2 2025-10-10T01:33:18.1964092Z  # via 2025-10-10T01:33:18.1964257Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1964484Z # cupy-cuda12x 2025-10-10T01:33:18.1964662Z fastsafetensors==0.1.10 2025-10-10T01:33:18.1964865Z  # via 2025-10-10T01:33:18.1965027Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1965271Z # -r requirements/test.in 2025-10-10T01:33:18.1965467Z filelock==3.16.1 2025-10-10T01:33:18.1965650Z  # via 2025-10-10T01:33:18.1965811Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1965997Z # blobfile 2025-10-10T01:33:18.1966138Z # datasets 2025-10-10T01:33:18.1966293Z # huggingface-hub 2025-10-10T01:33:18.1966455Z # ray 2025-10-10T01:33:18.1966589Z # torch 2025-10-10T01:33:18.1966739Z # transformers 2025-10-10T01:33:18.1966932Z # virtualenv 2025-10-10T01:33:18.1967095Z fiona==1.10.1 2025-10-10T01:33:18.1967268Z  # via 2025-10-10T01:33:18.1967536Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1967774Z # torchgeo 2025-10-10T01:33:18.1967944Z flask==3.1.1 2025-10-10T01:33:18.1968113Z  # via 2025-10-10T01:33:18.1968279Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1968503Z # mlflow 2025-10-10T01:33:18.1968664Z fonttools==4.55.0 2025-10-10T01:33:18.1968944Z  # via 2025-10-10T01:33:18.1969110Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1969338Z # matplotlib 2025-10-10T01:33:18.1969515Z fqdn==1.5.1 2025-10-10T01:33:18.1969680Z  # via 2025-10-10T01:33:18.1969842Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1970067Z # jsonschema 2025-10-10T01:33:18.1970237Z frozendict==2.4.6 2025-10-10T01:33:18.1970410Z  # via 2025-10-10T01:33:18.1970571Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1970787Z # einx 2025-10-10T01:33:18.1970939Z frozenlist==1.5.0 2025-10-10T01:33:18.1971116Z  # via 2025-10-10T01:33:18.1971279Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1971473Z # aiohttp 2025-10-10T01:33:18.1971652Z # aiosignal 2025-10-10T01:33:18.1971823Z fsspec==2024.9.0 2025-10-10T01:33:18.1972004Z  # via 2025-10-10T01:33:18.1972164Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1972351Z # datasets 2025-10-10T01:33:18.1972509Z # evaluate 2025-10-10T01:33:18.1972666Z # fastparquet 2025-10-10T01:33:18.1972833Z # huggingface-hub 2025-10-10T01:33:18.1973077Z # lightning 2025-10-10T01:33:18.1973248Z # pytorch-lightning 2025-10-10T01:33:18.1973458Z # torch 2025-10-10T01:33:18.1973614Z ftfy==6.3.1 2025-10-10T01:33:18.1973778Z  # via 2025-10-10T01:33:18.1973939Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1974176Z # open-clip-torch 2025-10-10T01:33:18.1974371Z genai-perf==0.0.8 2025-10-10T01:33:18.1974540Z  # via 2025-10-10T01:33:18.1974702Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1974944Z # -r requirements/test.in 2025-10-10T01:33:18.1975152Z genson==1.3.0 2025-10-10T01:33:18.1975317Z  # via 2025-10-10T01:33:18.1975488Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1975747Z # datamodel-code-generator 2025-10-10T01:33:18.1975966Z geopandas==1.0.1 2025-10-10T01:33:18.1976137Z  # via 2025-10-10T01:33:18.1976303Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1976527Z # terratorch 2025-10-10T01:33:18.1976694Z gitdb==4.0.12 2025-10-10T01:33:18.1976865Z  # via 2025-10-10T01:33:18.1977024Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1977245Z # gitpython 2025-10-10T01:33:18.1977410Z gitpython==3.1.44 2025-10-10T01:33:18.1977588Z  # via 2025-10-10T01:33:18.1977760Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1977992Z # mlflow-skinny 2025-10-10T01:33:18.1978177Z google-api-core==2.24.2 2025-10-10T01:33:18.1978371Z  # via 2025-10-10T01:33:18.1978535Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1978746Z # google-cloud-core 2025-10-10T01:33:18.1978932Z # google-cloud-storage 2025-10-10T01:33:18.1979151Z # opencensus 2025-10-10T01:33:18.1979320Z google-auth==2.40.2 2025-10-10T01:33:18.1979499Z  # via 2025-10-10T01:33:18.1979655Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1979866Z # databricks-sdk 2025-10-10T01:33:18.1980041Z # google-api-core 2025-10-10T01:33:18.1980218Z # google-cloud-core 2025-10-10T01:33:18.1980406Z # google-cloud-storage 2025-10-10T01:33:18.1980650Z # runai-model-streamer-gcs 2025-10-10T01:33:18.1980873Z google-cloud-core==2.4.3 2025-10-10T01:33:18.1981078Z  # via 2025-10-10T01:33:18.1981246Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1981495Z # google-cloud-storage 2025-10-10T01:33:18.1981712Z google-cloud-storage==3.4.0 2025-10-10T01:33:18.1981921Z  # via 2025-10-10T01:33:18.1982075Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1982322Z # runai-model-streamer-gcs 2025-10-10T01:33:18.1982536Z google-crc32c==1.7.1 2025-10-10T01:33:18.1982723Z  # via 2025-10-10T01:33:18.1982877Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1983080Z # google-cloud-storage 2025-10-10T01:33:18.1983319Z # google-resumable-media 2025-10-10T01:33:18.1983632Z google-resumable-media==2.7.2 2025-10-10T01:33:18.1983843Z  # via 2025-10-10T01:33:18.1984005Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1984246Z # google-cloud-storage 2025-10-10T01:33:18.1984462Z googleapis-common-protos==1.70.0 2025-10-10T01:33:18.1984680Z  # via 2025-10-10T01:33:18.1984840Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1985085Z # google-api-core 2025-10-10T01:33:18.1985270Z graphene==3.4.3 2025-10-10T01:33:18.1985442Z  # via 2025-10-10T01:33:18.1985601Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1985820Z # mlflow 2025-10-10T01:33:18.1985977Z graphql-core==3.2.6 2025-10-10T01:33:18.1986158Z  # via 2025-10-10T01:33:18.1986318Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1986509Z # graphene 2025-10-10T01:33:18.1986662Z # graphql-relay 2025-10-10T01:33:18.1986879Z # hypothesis-graphql 2025-10-10T01:33:18.1987079Z graphql-relay==3.2.0 2025-10-10T01:33:18.1987277Z  # via 2025-10-10T01:33:18.1987433Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1987673Z # graphene 2025-10-10T01:33:18.1987920Z greenlet==3.2.3 2025-10-10T01:33:18.1988103Z  # via 2025-10-10T01:33:18.1988262Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1988491Z # sqlalchemy 2025-10-10T01:33:18.1988659Z grpcio==1.71.0 2025-10-10T01:33:18.1988830Z  # via 2025-10-10T01:33:18.1988985Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1989205Z # ray 2025-10-10T01:33:18.1989366Z gunicorn==23.0.0 2025-10-10T01:33:18.1989542Z  # via 2025-10-10T01:33:18.1989697Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1989918Z # mlflow 2025-10-10T01:33:18.1990077Z h11==0.14.0 2025-10-10T01:33:18.1990247Z  # via 2025-10-10T01:33:18.1990402Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1990594Z # httpcore 2025-10-10T01:33:18.1990776Z # uvicorn 2025-10-10T01:33:18.1990943Z h5py==3.13.0 2025-10-10T01:33:18.1991117Z  # via 2025-10-10T01:33:18.1991277Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1991501Z # terratorch 2025-10-10T01:33:18.1991674Z harfile==0.3.0 2025-10-10T01:33:18.1991850Z  # via 2025-10-10T01:33:18.1992014Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1992237Z # schemathesis 2025-10-10T01:33:18.1992417Z hf-xet==1.1.7 2025-10-10T01:33:18.1992589Z  # via 2025-10-10T01:33:18.1992747Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1992975Z # huggingface-hub 2025-10-10T01:33:18.1993162Z hiredis==3.0.0 2025-10-10T01:33:18.1993333Z  # via 2025-10-10T01:33:18.1993488Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1993709Z # tensorizer 2025-10-10T01:33:18.1993874Z httpcore==1.0.6 2025-10-10T01:33:18.1994047Z  # via 2025-10-10T01:33:18.1994199Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1994414Z # httpx 2025-10-10T01:33:18.1994578Z httpx==0.27.2 2025-10-10T01:33:18.1994746Z  # via 2025-10-10T01:33:18.1994897Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1995105Z # -r requirements/test.in 2025-10-10T01:33:18.1995331Z # schemathesis 2025-10-10T01:33:18.1995520Z huggingface-hub==0.34.3 2025-10-10T01:33:18.1995711Z  # via 2025-10-10T01:33:18.1995872Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.1996064Z # accelerate 2025-10-10T01:33:18.1996879Z # datasets 2025-10-10T01:33:18.1997153Z # evaluate 2025-10-10T01:33:18.1997430Z # open-clip-torch 2025-10-10T01:33:18.1997716Z # peft 2025-10-10T01:33:18.1997993Z # segmentation-models-pytorch 2025-10-10T01:33:18.1998262Z # sentence-transformers 2025-10-10T01:33:18.1998458Z # terratorch 2025-10-10T01:33:18.1998618Z # timm 2025-10-10T01:33:18.1998776Z # tokenizers 2025-10-10T01:33:18.1998923Z # transformers 2025-10-10T01:33:18.1999247Z # vocos 2025-10-10T01:33:18.1999637Z humanize==4.11.0 2025-10-10T01:33:18.1999828Z  # via 2025-10-10T01:33:18.2000011Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2000279Z # runai-model-streamer 2025-10-10T01:33:18.2000484Z hydra-core==1.3.2 2025-10-10T01:33:18.2000660Z  # via 2025-10-10T01:33:18.2000829Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2001023Z # lightly 2025-10-10T01:33:18.2001211Z # lightning 2025-10-10T01:33:18.2001390Z hypothesis==6.131.0 2025-10-10T01:33:18.2001576Z  # via 2025-10-10T01:33:18.2001744Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2001951Z # hypothesis-graphql 2025-10-10T01:33:18.2002143Z # hypothesis-jsonschema 2025-10-10T01:33:18.2002373Z # schemathesis 2025-10-10T01:33:18.2002571Z hypothesis-graphql==0.11.1 2025-10-10T01:33:18.2002780Z  # via 2025-10-10T01:33:18.2002949Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2003192Z # schemathesis 2025-10-10T01:33:18.2003394Z hypothesis-jsonschema==0.23.1 2025-10-10T01:33:18.2003618Z  # via 2025-10-10T01:33:18.2003872Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2004368Z # schemathesis 2025-10-10T01:33:18.2004554Z idna==3.10 2025-10-10T01:33:18.2004727Z  # via 2025-10-10T01:33:18.2004884Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2005081Z # anyio 2025-10-10T01:33:18.2005243Z # email-validator 2025-10-10T01:33:18.2005411Z # httpx 2025-10-10T01:33:18.2005551Z # jsonschema 2025-10-10T01:33:18.2005704Z # requests 2025-10-10T01:33:18.2005885Z # yarl 2025-10-10T01:33:18.2006045Z imageio==2.37.0 2025-10-10T01:33:18.2006224Z  # via 2025-10-10T01:33:18.2006390Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2006618Z # scikit-image 2025-10-10T01:33:18.2006812Z importlib-metadata==8.7.0 2025-10-10T01:33:18.2007011Z  # via 2025-10-10T01:33:18.2007163Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2007355Z # mlflow-skinny 2025-10-10T01:33:18.2007569Z # opentelemetry-api 2025-10-10T01:33:18.2007922Z importlib-resources==6.5.2 2025-10-10T01:33:18.2008160Z  # via 2025-10-10T01:33:18.2008332Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2008562Z # typeshed-client 2025-10-10T01:33:18.2008752Z inflect==5.6.2 2025-10-10T01:33:18.2008921Z  # via 2025-10-10T01:33:18.2009071Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2009318Z # datamodel-code-generator 2025-10-10T01:33:18.2009523Z iniconfig==2.0.0 2025-10-10T01:33:18.2009709Z  # via 2025-10-10T01:33:18.2009863Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2010076Z # pytest 2025-10-10T01:33:18.2010254Z isoduration==20.11.0 2025-10-10T01:33:18.2010573Z  # via 2025-10-10T01:33:18.2010744Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2010970Z # jsonschema 2025-10-10T01:33:18.2011129Z isort==5.13.2 2025-10-10T01:33:18.2011304Z  # via 2025-10-10T01:33:18.2011474Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2011721Z # datamodel-code-generator 2025-10-10T01:33:18.2011939Z itsdangerous==2.2.0 2025-10-10T01:33:18.2012225Z  # via 2025-10-10T01:33:18.2012421Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2012651Z # flask 2025-10-10T01:33:18.2012799Z jinja2==3.1.6 2025-10-10T01:33:18.2012968Z  # via 2025-10-10T01:33:18.2013125Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2013332Z # datamodel-code-generator 2025-10-10T01:33:18.2013518Z # flask 2025-10-10T01:33:18.2013662Z # mlflow 2025-10-10T01:33:18.2013834Z # torch 2025-10-10T01:33:18.2013989Z jiwer==3.0.5 2025-10-10T01:33:18.2014152Z  # via 2025-10-10T01:33:18.2014310Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2014552Z # -r requirements/test.in 2025-10-10T01:33:18.2014752Z jmespath==1.0.1 2025-10-10T01:33:18.2014928Z  # via 2025-10-10T01:33:18.2015093Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2015402Z # boto3 2025-10-10T01:33:18.2015578Z # botocore 2025-10-10T01:33:18.2015750Z joblib==1.4.2 2025-10-10T01:33:18.2015929Z  # via 2025-10-10T01:33:18.2016090Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2016274Z # librosa 2025-10-10T01:33:18.2016424Z # nltk 2025-10-10T01:33:18.2016609Z # scikit-learn 2025-10-10T01:33:18.2016792Z jsonargparse==4.35.0 2025-10-10T01:33:18.2016979Z  # via 2025-10-10T01:33:18.2017139Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2017330Z # lightning 2025-10-10T01:33:18.2017521Z # terratorch 2025-10-10T01:33:18.2017684Z jsonlines==4.0.0 2025-10-10T01:33:18.2017862Z  # via 2025-10-10T01:33:18.2018019Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2018235Z # lm-eval 2025-10-10T01:33:18.2018391Z jsonpointer==3.0.0 2025-10-10T01:33:18.2018576Z  # via 2025-10-10T01:33:18.2018736Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2018964Z # jsonschema 2025-10-10T01:33:18.2019130Z jsonschema==4.23.0 2025-10-10T01:33:18.2019307Z  # via 2025-10-10T01:33:18.2019544Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2019758Z # hypothesis-jsonschema 2025-10-10T01:33:18.2019950Z # mistral-common 2025-10-10T01:33:18.2020112Z # ray 2025-10-10T01:33:18.2020291Z # schemathesis 2025-10-10T01:33:18.2020480Z jsonschema-specifications==2024.10.1 2025-10-10T01:33:18.2020710Z  # via 2025-10-10T01:33:18.2020940Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2021220Z # jsonschema 2025-10-10T01:33:18.2021381Z junit-xml==1.9 2025-10-10T01:33:18.2021550Z  # via 2025-10-10T01:33:18.2021717Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2021950Z # schemathesis 2025-10-10T01:33:18.2022115Z kaleido==0.2.1 2025-10-10T01:33:18.2022287Z  # via 2025-10-10T01:33:18.2022445Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2022685Z # genai-perf 2025-10-10T01:33:18.2022847Z kiwisolver==1.4.7 2025-10-10T01:33:18.2023027Z  # via 2025-10-10T01:33:18.2023190Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2023414Z # matplotlib 2025-10-10T01:33:18.2023571Z kornia==0.8.1 2025-10-10T01:33:18.2023739Z  # via 2025-10-10T01:33:18.2023898Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2024116Z # torchgeo 2025-10-10T01:33:18.2024275Z kornia-rs==0.1.9 2025-10-10T01:33:18.2024452Z  # via 2025-10-10T01:33:18.2024609Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2024825Z # kornia 2025-10-10T01:33:18.2024981Z lazy-loader==0.4 2025-10-10T01:33:18.2025159Z  # via 2025-10-10T01:33:18.2025318Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2025520Z # librosa 2025-10-10T01:33:18.2025700Z # scikit-image 2025-10-10T01:33:18.2025874Z libnacl==2.1.0 2025-10-10T01:33:18.2026045Z  # via 2025-10-10T01:33:18.2026196Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2026436Z # tensorizer 2025-10-10T01:33:18.2026601Z librosa==0.10.2.post1 2025-10-10T01:33:18.2026790Z  # via 2025-10-10T01:33:18.2026946Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2027185Z # -r requirements/test.in 2025-10-10T01:33:18.2027386Z lightly==1.5.20 2025-10-10T01:33:18.2027557Z  # via 2025-10-10T01:33:18.2027706Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2027894Z # terratorch 2025-10-10T01:33:18.2028086Z # torchgeo 2025-10-10T01:33:18.2028252Z lightly-utils==0.0.2 2025-10-10T01:33:18.2028430Z  # via 2025-10-10T01:33:18.2028589Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2028805Z # lightly 2025-10-10T01:33:18.2028969Z lightning==2.5.1.post0 2025-10-10T01:33:18.2029147Z  # via 2025-10-10T01:33:18.2029304Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2029495Z # terratorch 2025-10-10T01:33:18.2029679Z # torchgeo 2025-10-10T01:33:18.2029950Z lightning-utilities==0.14.3 2025-10-10T01:33:18.2030172Z  # via 2025-10-10T01:33:18.2030347Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2030631Z # lightning 2025-10-10T01:33:18.2030796Z # pytorch-lightning 2025-10-10T01:33:18.2031011Z # torchmetrics 2025-10-10T01:33:18.2031197Z llvmlite==0.44.0 2025-10-10T01:33:18.2031375Z  # via 2025-10-10T01:33:18.2031529Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2031752Z # numba 2025-10-10T01:33:18.2032154Z lm-eval @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@206b7722158f58c35b7ffcd53b035fdbdda5126d 2025-10-10T01:33:18.2032591Z  # via 2025-10-10T01:33:18.2032749Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2032994Z # -r requirements/test.in 2025-10-10T01:33:18.2033195Z lxml==5.3.0 2025-10-10T01:33:18.2033360Z  # via 2025-10-10T01:33:18.2033514Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2033703Z # blobfile 2025-10-10T01:33:18.2033893Z # sacrebleu 2025-10-10T01:33:18.2034052Z mako==1.3.10 2025-10-10T01:33:18.2034230Z  # via 2025-10-10T01:33:18.2034475Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2034705Z # alembic 2025-10-10T01:33:18.2034863Z markdown==3.8.2 2025-10-10T01:33:18.2035051Z  # via 2025-10-10T01:33:18.2035210Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2035427Z # mlflow 2025-10-10T01:33:18.2035586Z markdown-it-py==3.0.0 2025-10-10T01:33:18.2035774Z  # via 2025-10-10T01:33:18.2035931Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2036143Z # rich 2025-10-10T01:33:18.2036291Z markupsafe==3.0.1 2025-10-10T01:33:18.2036530Z  # via 2025-10-10T01:33:18.2036826Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2037165Z # flask 2025-10-10T01:33:18.2037385Z # jinja2 2025-10-10T01:33:18.2037532Z # mako 2025-10-10T01:33:18.2037722Z # werkzeug 2025-10-10T01:33:18.2037892Z matplotlib==3.9.2 2025-10-10T01:33:18.2038061Z  # via 2025-10-10T01:33:18.2038220Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2038434Z # -r requirements/test.in 2025-10-10T01:33:18.2038624Z # lightning 2025-10-10T01:33:18.2038765Z # mlflow 2025-10-10T01:33:18.2038909Z # pycocotools 2025-10-10T01:33:18.2039189Z # torchgeo 2025-10-10T01:33:18.2039356Z mbstrdecoder==1.1.3 2025-10-10T01:33:18.2039551Z  # via 2025-10-10T01:33:18.2039715Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2039908Z # dataproperty 2025-10-10T01:33:18.2040066Z # pytablewriter 2025-10-10T01:33:18.2040256Z # typepy 2025-10-10T01:33:18.2040423Z mdurl==0.1.2 2025-10-10T01:33:18.2040591Z  # via 2025-10-10T01:33:18.2040744Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2040976Z # markdown-it-py 2025-10-10T01:33:18.2041164Z mistral-common==1.8.2 2025-10-10T01:33:18.2041347Z  # via 2025-10-10T01:33:18.2041515Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2041757Z # -r requirements/test.in 2025-10-10T01:33:18.2041967Z mlflow==2.22.0 2025-10-10T01:33:18.2042139Z  # via 2025-10-10T01:33:18.2042295Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2042517Z # terratorch 2025-10-10T01:33:18.2042692Z mlflow-skinny==2.22.0 2025-10-10T01:33:18.2042889Z  # via 2025-10-10T01:33:18.2043046Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2043271Z # mlflow 2025-10-10T01:33:18.2043436Z more-itertools==10.5.0 2025-10-10T01:33:18.2043629Z  # via 2025-10-10T01:33:18.2043785Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2044004Z # lm-eval 2025-10-10T01:33:18.2044167Z mpmath==1.3.0 2025-10-10T01:33:18.2044337Z  # via 2025-10-10T01:33:18.2044485Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2044695Z # sympy 2025-10-10T01:33:18.2044848Z msgpack==1.1.0 2025-10-10T01:33:18.2045010Z  # via 2025-10-10T01:33:18.2045277Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2045475Z # librosa 2025-10-10T01:33:18.2045650Z # ray 2025-10-10T01:33:18.2045798Z mteb==1.38.11 2025-10-10T01:33:18.2045965Z  # via 2025-10-10T01:33:18.2046116Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2046351Z # -r requirements/test.in 2025-10-10T01:33:18.2046556Z multidict==6.1.0 2025-10-10T01:33:18.2046730Z  # via 2025-10-10T01:33:18.2046885Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2047067Z # aiohttp 2025-10-10T01:33:18.2047232Z # yarl 2025-10-10T01:33:18.2047388Z multiprocess==0.70.16 2025-10-10T01:33:18.2047569Z  # via 2025-10-10T01:33:18.2047724Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2047906Z # datasets 2025-10-10T01:33:18.2048077Z # evaluate 2025-10-10T01:33:18.2048245Z munch==4.0.0 2025-10-10T01:33:18.2048411Z  # via 2025-10-10T01:33:18.2048555Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2048810Z # pretrainedmodels 2025-10-10T01:33:18.2049007Z mypy-extensions==1.0.0 2025-10-10T01:33:18.2049191Z  # via 2025-10-10T01:33:18.2049440Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2049668Z # black 2025-10-10T01:33:18.2049821Z networkx==3.2.1 2025-10-10T01:33:18.2049987Z  # via 2025-10-10T01:33:18.2050143Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2050336Z # scikit-image 2025-10-10T01:33:18.2050520Z # torch 2025-10-10T01:33:18.2050664Z nltk==3.9.1 2025-10-10T01:33:18.2050831Z  # via 2025-10-10T01:33:18.2050989Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2051213Z # rouge-score 2025-10-10T01:33:18.2051381Z num2words==0.5.14 2025-10-10T01:33:18.2051556Z  # via 2025-10-10T01:33:18.2051726Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2051968Z # -r requirements/test.in 2025-10-10T01:33:18.2052158Z numba==0.61.2 2025-10-10T01:33:18.2052337Z  # via 2025-10-10T01:33:18.2052504Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2052702Z # -r requirements/test.in 2025-10-10T01:33:18.2052917Z # librosa 2025-10-10T01:33:18.2053080Z numexpr==2.10.1 2025-10-10T01:33:18.2053249Z  # via 2025-10-10T01:33:18.2053402Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2053698Z # lm-eval 2025-10-10T01:33:18.2053985Z numpy==1.26.4 2025-10-10T01:33:18.2054300Z  # via 2025-10-10T01:33:18.2054527Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2054728Z # -r requirements/test.in 2025-10-10T01:33:18.2054911Z # accelerate 2025-10-10T01:33:18.2055064Z # albucore 2025-10-10T01:33:18.2055214Z # albumentations 2025-10-10T01:33:18.2055385Z # bitsandbytes 2025-10-10T01:33:18.2055537Z # bm25s 2025-10-10T01:33:18.2055680Z # contourpy 2025-10-10T01:33:18.2055826Z # cupy-cuda12x 2025-10-10T01:33:18.2055978Z # datasets 2025-10-10T01:33:18.2056127Z # decord 2025-10-10T01:33:18.2056278Z # einx 2025-10-10T01:33:18.2056412Z # encodec 2025-10-10T01:33:18.2056557Z # evaluate 2025-10-10T01:33:18.2056709Z # fastparquet 2025-10-10T01:33:18.2056876Z # genai-perf 2025-10-10T01:33:18.2057023Z # geopandas 2025-10-10T01:33:18.2057174Z # h5py 2025-10-10T01:33:18.2057313Z # imageio 2025-10-10T01:33:18.2057454Z # librosa 2025-10-10T01:33:18.2057590Z # lightly 2025-10-10T01:33:18.2057741Z # lightly-utils 2025-10-10T01:33:18.2057915Z # matplotlib 2025-10-10T01:33:18.2058073Z # mistral-common 2025-10-10T01:33:18.2058228Z # mlflow 2025-10-10T01:33:18.2058370Z # mteb 2025-10-10T01:33:18.2058509Z # numba 2025-10-10T01:33:18.2058642Z # numexpr 2025-10-10T01:33:18.2058802Z # opencv-python-headless 2025-10-10T01:33:18.2058993Z # pandas 2025-10-10T01:33:18.2059139Z # patsy 2025-10-10T01:33:18.2059277Z # peft 2025-10-10T01:33:18.2059429Z # pycocotools 2025-10-10T01:33:18.2059690Z # pyogrio 2025-10-10T01:33:18.2059837Z # rasterio 2025-10-10T01:33:18.2059992Z # rioxarray 2025-10-10T01:33:18.2060155Z # rouge-score 2025-10-10T01:33:18.2060322Z # runai-model-streamer 2025-10-10T01:33:18.2060508Z # sacrebleu 2025-10-10T01:33:18.2060655Z # scikit-image 2025-10-10T01:33:18.2060812Z # scikit-learn 2025-10-10T01:33:18.2060968Z # scipy 2025-10-10T01:33:18.2061129Z # segmentation-models-pytorch 2025-10-10T01:33:18.2061326Z # shapely 2025-10-10T01:33:18.2061467Z # soxr 2025-10-10T01:33:18.2061606Z # statsmodels 2025-10-10T01:33:18.2061758Z # tensorboardx 2025-10-10T01:33:18.2061906Z # tensorizer 2025-10-10T01:33:18.2062052Z # tifffile 2025-10-10T01:33:18.2062198Z # torchgeo 2025-10-10T01:33:18.2062343Z # torchmetrics 2025-10-10T01:33:18.2062496Z # torchvision 2025-10-10T01:33:18.2062646Z # transformers 2025-10-10T01:33:18.2062811Z # tritonclient 2025-10-10T01:33:18.2062964Z # vocos 2025-10-10T01:33:18.2063148Z # xarray 2025-10-10T01:33:18.2063302Z omegaconf==2.3.0 2025-10-10T01:33:18.2063556Z  # via 2025-10-10T01:33:18.2063725Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2063918Z # hydra-core 2025-10-10T01:33:18.2064102Z # lightning 2025-10-10T01:33:18.2064274Z open-clip-torch==2.32.0 2025-10-10T01:33:18.2064504Z  # via -r requirements/test.in 2025-10-10T01:33:18.2064725Z opencensus==0.11.4 2025-10-10T01:33:18.2064900Z  # via 2025-10-10T01:33:18.2065054Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2065265Z # ray 2025-10-10T01:33:18.2065431Z opencensus-context==0.1.3 2025-10-10T01:33:18.2065634Z  # via 2025-10-10T01:33:18.2065797Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2066190Z # opencensus 2025-10-10T01:33:18.2066439Z opencv-python-headless==4.11.0.86 2025-10-10T01:33:18.2066742Z  # via 2025-10-10T01:33:18.2067043Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2067341Z # -r requirements/test.in 2025-10-10T01:33:18.2067531Z # albucore 2025-10-10T01:33:18.2067698Z # albumentations 2025-10-10T01:33:18.2067926Z # mistral-common 2025-10-10T01:33:18.2068111Z opentelemetry-api==1.35.0 2025-10-10T01:33:18.2068312Z  # via 2025-10-10T01:33:18.2068475Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2068672Z # mlflow-skinny 2025-10-10T01:33:18.2068864Z # opentelemetry-exporter-prometheus 2025-10-10T01:33:18.2069095Z # opentelemetry-sdk 2025-10-10T01:33:18.2069355Z # opentelemetry-semantic-conventions 2025-10-10T01:33:18.2069619Z opentelemetry-exporter-prometheus==0.56b0 2025-10-10T01:33:18.2069859Z  # via 2025-10-10T01:33:18.2070015Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2070229Z # ray 2025-10-10T01:33:18.2070389Z opentelemetry-proto==1.36.0 2025-10-10T01:33:18.2070590Z  # via 2025-10-10T01:33:18.2070742Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2070963Z # ray 2025-10-10T01:33:18.2071116Z opentelemetry-sdk==1.35.0 2025-10-10T01:33:18.2071317Z  # via 2025-10-10T01:33:18.2071483Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2071680Z # mlflow-skinny 2025-10-10T01:33:18.2071863Z # opentelemetry-exporter-prometheus 2025-10-10T01:33:18.2072102Z # ray 2025-10-10T01:33:18.2072283Z opentelemetry-semantic-conventions==0.56b0 2025-10-10T01:33:18.2072527Z  # via 2025-10-10T01:33:18.2072677Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2072910Z # opentelemetry-sdk 2025-10-10T01:33:18.2073099Z packaging==24.2 2025-10-10T01:33:18.2073272Z  # via 2025-10-10T01:33:18.2073423Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2073613Z # accelerate 2025-10-10T01:33:18.2073762Z # black 2025-10-10T01:33:18.2073935Z # datamodel-code-generator 2025-10-10T01:33:18.2074120Z # datasets 2025-10-10T01:33:18.2074264Z # evaluate 2025-10-10T01:33:18.2074528Z # fastparquet 2025-10-10T01:33:18.2074687Z # geopandas 2025-10-10T01:33:18.2074832Z # gunicorn 2025-10-10T01:33:18.2075003Z # huggingface-hub 2025-10-10T01:33:18.2075177Z # hydra-core 2025-10-10T01:33:18.2075323Z # kornia 2025-10-10T01:33:18.2075463Z # lazy-loader 2025-10-10T01:33:18.2075617Z # lightning 2025-10-10T01:33:18.2075776Z # lightning-utilities 2025-10-10T01:33:18.2075966Z # matplotlib 2025-10-10T01:33:18.2076126Z # mlflow-skinny 2025-10-10T01:33:18.2076288Z # peft 2025-10-10T01:33:18.2076434Z # plotly 2025-10-10T01:33:18.2076575Z # pooch 2025-10-10T01:33:18.2076722Z # pyogrio 2025-10-10T01:33:18.2076868Z # pytest 2025-10-10T01:33:18.2077031Z # pytest-rerunfailures 2025-10-10T01:33:18.2077220Z # pytorch-lightning 2025-10-10T01:33:18.2077399Z # ray 2025-10-10T01:33:18.2077543Z # rioxarray 2025-10-10T01:33:18.2077698Z # scikit-image 2025-10-10T01:33:18.2077862Z # statsmodels 2025-10-10T01:33:18.2078020Z # tensorboardx 2025-10-10T01:33:18.2078178Z # torchmetrics 2025-10-10T01:33:18.2078418Z # transformers 2025-10-10T01:33:18.2078575Z # typepy 2025-10-10T01:33:18.2078751Z # xarray 2025-10-10T01:33:18.2078905Z pandas==2.2.3 2025-10-10T01:33:18.2079161Z  # via 2025-10-10T01:33:18.2079329Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2079520Z # datasets 2025-10-10T01:33:18.2079665Z # evaluate 2025-10-10T01:33:18.2079807Z # fastparquet 2025-10-10T01:33:18.2079961Z # genai-perf 2025-10-10T01:33:18.2080124Z # geopandas 2025-10-10T01:33:18.2080271Z # mlflow 2025-10-10T01:33:18.2090398Z # statsmodels 2025-10-10T01:33:18.2090613Z # torchgeo 2025-10-10T01:33:18.2090851Z # xarray 2025-10-10T01:33:18.2091024Z pathspec==0.12.1 2025-10-10T01:33:18.2091225Z  # via 2025-10-10T01:33:18.2091403Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2091654Z # black 2025-10-10T01:33:18.2091838Z pathvalidate==3.2.1 2025-10-10T01:33:18.2092025Z  # via 2025-10-10T01:33:18.2092185Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2092429Z # pytablewriter 2025-10-10T01:33:18.2092606Z patsy==1.0.1 2025-10-10T01:33:18.2092780Z  # via 2025-10-10T01:33:18.2092945Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2093181Z # statsmodels 2025-10-10T01:33:18.2093355Z peft==0.16.0 2025-10-10T01:33:18.2093524Z  # via 2025-10-10T01:33:18.2093687Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2093893Z # -r requirements/test.in 2025-10-10T01:33:18.2094108Z # lm-eval 2025-10-10T01:33:18.2094269Z pillow==10.4.0 2025-10-10T01:33:18.2094431Z  # via 2025-10-10T01:33:18.2094591Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2094783Z # genai-perf 2025-10-10T01:33:18.2094948Z # imageio 2025-10-10T01:33:18.2095097Z # lightly-utils 2025-10-10T01:33:18.2095274Z # matplotlib 2025-10-10T01:33:18.2095436Z # mistral-common 2025-10-10T01:33:18.2095606Z # scikit-image 2025-10-10T01:33:18.2095782Z # segmentation-models-pytorch 2025-10-10T01:33:18.2096000Z # sentence-transformers 2025-10-10T01:33:18.2096471Z # torchgeo 2025-10-10T01:33:18.2096685Z # torchvision 2025-10-10T01:33:18.2096862Z platformdirs==4.3.6 2025-10-10T01:33:18.2097056Z  # via 2025-10-10T01:33:18.2097221Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2097408Z # black 2025-10-10T01:33:18.2097556Z # pooch 2025-10-10T01:33:18.2097735Z # virtualenv 2025-10-10T01:33:18.2097907Z plotly==5.24.1 2025-10-10T01:33:18.2098070Z  # via 2025-10-10T01:33:18.2098228Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2098454Z # genai-perf 2025-10-10T01:33:18.2098629Z pluggy==1.5.0 2025-10-10T01:33:18.2098793Z  # via 2025-10-10T01:33:18.2098963Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2099351Z # pytest 2025-10-10T01:33:18.2099545Z # pytest-cov 2025-10-10T01:33:18.2099706Z polars==1.29.0 2025-10-10T01:33:18.2099882Z  # via 2025-10-10T01:33:18.2100049Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2100267Z # mteb 2025-10-10T01:33:18.2100425Z pooch==1.8.2 2025-10-10T01:33:18.2100598Z  # via 2025-10-10T01:33:18.2100768Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2100988Z # librosa 2025-10-10T01:33:18.2101149Z portalocker==2.10.1 2025-10-10T01:33:18.2101338Z  # via 2025-10-10T01:33:18.2101497Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2101717Z # sacrebleu 2025-10-10T01:33:18.2101880Z pqdm==0.2.0 2025-10-10T01:33:18.2102057Z  # via 2025-10-10T01:33:18.2102226Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2102473Z # -r requirements/test.in 2025-10-10T01:33:18.2102869Z pretrainedmodels==0.7.4 2025-10-10T01:33:18.2103220Z  # via 2025-10-10T01:33:18.2103401Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2103656Z # segmentation-models-pytorch 2025-10-10T01:33:18.2104020Z prometheus-client==0.22.0 2025-10-10T01:33:18.2104243Z  # via 2025-10-10T01:33:18.2104406Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2104618Z # opentelemetry-exporter-prometheus 2025-10-10T01:33:18.2104859Z # ray 2025-10-10T01:33:18.2105016Z propcache==0.2.0 2025-10-10T01:33:18.2105194Z  # via 2025-10-10T01:33:18.2105350Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2105581Z # yarl 2025-10-10T01:33:18.2105743Z proto-plus==1.26.1 2025-10-10T01:33:18.2105923Z  # via 2025-10-10T01:33:18.2106073Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2106305Z # google-api-core 2025-10-10T01:33:18.2106491Z protobuf==5.28.3 2025-10-10T01:33:18.2106667Z  # via 2025-10-10T01:33:18.2106818Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2107012Z # google-api-core 2025-10-10T01:33:18.2107217Z # googleapis-common-protos 2025-10-10T01:33:18.2107416Z # mlflow-skinny 2025-10-10T01:33:18.2107590Z # opentelemetry-proto 2025-10-10T01:33:18.2107816Z # proto-plus 2025-10-10T01:33:18.2108102Z # ray 2025-10-10T01:33:18.2108331Z # tensorboardx 2025-10-10T01:33:18.2108548Z # tensorizer 2025-10-10T01:33:18.2108716Z psutil==6.1.0 2025-10-10T01:33:18.2108895Z  # via 2025-10-10T01:33:18.2109055Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2109241Z # accelerate 2025-10-10T01:33:18.2109307Z # peft 2025-10-10T01:33:18.2109411Z # tensorizer 2025-10-10T01:33:18.2109475Z py==1.11.0 2025-10-10T01:33:18.2109601Z  # via 2025-10-10T01:33:18.2109737Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2109920Z # pytest-forked 2025-10-10T01:33:18.2109988Z py-spy==0.4.0 2025-10-10T01:33:18.2110071Z  # via 2025-10-10T01:33:18.2110154Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2110248Z # ray 2025-10-10T01:33:18.2110309Z pyarrow==18.0.0 2025-10-10T01:33:18.2110395Z  # via 2025-10-10T01:33:18.2110475Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2110547Z # datasets 2025-10-10T01:33:18.2110609Z # genai-perf 2025-10-10T01:33:18.2110698Z # mlflow 2025-10-10T01:33:18.2110764Z pyasn1==0.6.1 2025-10-10T01:33:18.2110842Z  # via 2025-10-10T01:33:18.2110917Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2110980Z # pyasn1-modules 2025-10-10T01:33:18.2111061Z # rsa 2025-10-10T01:33:18.2111135Z pyasn1-modules==0.4.2 2025-10-10T01:33:18.2111213Z  # via 2025-10-10T01:33:18.2111288Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2111388Z # google-auth 2025-10-10T01:33:18.2111451Z pybind11==2.13.6 2025-10-10T01:33:18.2111546Z  # via 2025-10-10T01:33:18.2111620Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2111705Z # lm-eval 2025-10-10T01:33:18.2111771Z pycocotools==2.0.8 2025-10-10T01:33:18.2111994Z  # via 2025-10-10T01:33:18.2112069Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2112167Z # terratorch 2025-10-10T01:33:18.2112230Z pycountry==24.6.1 2025-10-10T01:33:18.2112307Z  # via 2025-10-10T01:33:18.2112382Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2112490Z # pydantic-extra-types 2025-10-10T01:33:18.2112556Z pycparser==2.22 2025-10-10T01:33:18.2112634Z  # via 2025-10-10T01:33:18.2112704Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2112789Z # cffi 2025-10-10T01:33:18.2112856Z pycryptodomex==3.22.0 2025-10-10T01:33:18.2112934Z  # via 2025-10-10T01:33:18.2113009Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2113094Z # blobfile 2025-10-10T01:33:18.2113159Z pydantic==2.11.7 2025-10-10T01:33:18.2113236Z  # via 2025-10-10T01:33:18.2113304Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2113380Z # -r requirements/test.in 2025-10-10T01:33:18.2113457Z # albumentations 2025-10-10T01:33:18.2113537Z # datamodel-code-generator 2025-10-10T01:33:18.2113596Z # fastapi 2025-10-10T01:33:18.2113733Z # lightly 2025-10-10T01:33:18.2113809Z # mistral-common 2025-10-10T01:33:18.2113869Z # mlflow-skinny 2025-10-10T01:33:18.2113925Z # mteb 2025-10-10T01:33:18.2114005Z # pydantic-extra-types 2025-10-10T01:33:18.2114090Z # ray 2025-10-10T01:33:18.2114170Z pydantic-core==2.33.2 2025-10-10T01:33:18.2114249Z  # via 2025-10-10T01:33:18.2114320Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2114411Z # pydantic 2025-10-10T01:33:18.2114487Z pydantic-extra-types==2.10.5 2025-10-10T01:33:18.2114571Z  # via 2025-10-10T01:33:18.2114642Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2114739Z # mistral-common 2025-10-10T01:33:18.2114805Z pygments==2.18.0 2025-10-10T01:33:18.2114883Z  # via 2025-10-10T01:33:18.2114953Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2115052Z # rich 2025-10-10T01:33:18.2115114Z pyogrio==0.11.0 2025-10-10T01:33:18.2115199Z  # via 2025-10-10T01:33:18.2115277Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2115365Z # geopandas 2025-10-10T01:33:18.2115438Z pyparsing==3.2.0 2025-10-10T01:33:18.2115518Z  # via 2025-10-10T01:33:18.2115591Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2115650Z # matplotlib 2025-10-10T01:33:18.2115736Z # rasterio 2025-10-10T01:33:18.2115802Z pyproj==3.7.1 2025-10-10T01:33:18.2115882Z  # via 2025-10-10T01:33:18.2115955Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2116019Z # geopandas 2025-10-10T01:33:18.2116077Z # rioxarray 2025-10-10T01:33:18.2116168Z # torchgeo 2025-10-10T01:33:18.2116236Z pyrate-limiter==3.7.0 2025-10-10T01:33:18.2116313Z  # via 2025-10-10T01:33:18.2116388Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2116482Z # schemathesis 2025-10-10T01:33:18.2116555Z pystemmer==3.0.0 2025-10-10T01:33:18.2116637Z  # via 2025-10-10T01:33:18.2116716Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2116812Z # mteb 2025-10-10T01:33:18.2116878Z pytablewriter==1.2.0 2025-10-10T01:33:18.2116954Z  # via 2025-10-10T01:33:18.2117028Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2117115Z # lm-eval 2025-10-10T01:33:18.2117189Z pytest==8.3.5 2025-10-10T01:33:18.2117332Z  # via 2025-10-10T01:33:18.2117464Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2117595Z # -r requirements/test.in 2025-10-10T01:33:18.2117684Z # buildkite-test-collector 2025-10-10T01:33:18.2117750Z # genai-perf 2025-10-10T01:33:18.2117840Z # pytest-asyncio 2025-10-10T01:33:18.2117949Z # pytest-cov 2025-10-10T01:33:18.2118068Z # pytest-forked 2025-10-10T01:33:18.2118159Z # pytest-mock 2025-10-10T01:33:18.2118235Z # pytest-rerunfailures 2025-10-10T01:33:18.2118300Z # pytest-shard 2025-10-10T01:33:18.2118478Z # pytest-subtests 2025-10-10T01:33:18.2118547Z # pytest-timeout 2025-10-10T01:33:18.2118614Z # schemathesis 2025-10-10T01:33:18.2118722Z # terratorch 2025-10-10T01:33:18.2118803Z pytest-asyncio==0.24.0 2025-10-10T01:33:18.2118884Z  # via 2025-10-10T01:33:18.2118959Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2119131Z # -r requirements/test.in 2025-10-10T01:33:18.2119196Z pytest-cov==6.3.0 2025-10-10T01:33:18.2119288Z  # via 2025-10-10T01:33:18.2119358Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2119465Z # -r requirements/test.in 2025-10-10T01:33:18.2119534Z pytest-forked==1.6.0 2025-10-10T01:33:18.2119612Z  # via 2025-10-10T01:33:18.2119689Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2119790Z # -r requirements/test.in 2025-10-10T01:33:18.2119852Z pytest-mock==3.14.0 2025-10-10T01:33:18.2119933Z  # via 2025-10-10T01:33:18.2120002Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2120103Z # genai-perf 2025-10-10T01:33:18.2120176Z pytest-rerunfailures==14.0 2025-10-10T01:33:18.2120339Z  # via 2025-10-10T01:33:18.2120420Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2120537Z # -r requirements/test.in 2025-10-10T01:33:18.2120601Z pytest-shard==0.1.2 2025-10-10T01:33:18.2120685Z  # via 2025-10-10T01:33:18.2120754Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2120862Z # -r requirements/test.in 2025-10-10T01:33:18.2120929Z pytest-subtests==0.14.1 2025-10-10T01:33:18.2121007Z  # via 2025-10-10T01:33:18.2121122Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2121303Z # schemathesis 2025-10-10T01:33:18.2121412Z pytest-timeout==2.3.1 2025-10-10T01:33:18.2121502Z  # via 2025-10-10T01:33:18.2121575Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2121687Z # -r requirements/test.in 2025-10-10T01:33:18.2121756Z python-box==7.3.2 2025-10-10T01:33:18.2121843Z  # via 2025-10-10T01:33:18.2121922Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2122016Z # terratorch 2025-10-10T01:33:18.2122094Z python-dateutil==2.9.0.post0 2025-10-10T01:33:18.2122171Z  # via 2025-10-10T01:33:18.2122253Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2122316Z # arrow 2025-10-10T01:33:18.2122375Z # botocore 2025-10-10T01:33:18.2122439Z # graphene 2025-10-10T01:33:18.2122494Z # lightly 2025-10-10T01:33:18.2122553Z # matplotlib 2025-10-10T01:33:18.2122617Z # pandas 2025-10-10T01:33:18.2122705Z # typepy 2025-10-10T01:33:18.2122774Z python-rapidjson==1.20 2025-10-10T01:33:18.2122869Z  # via 2025-10-10T01:33:18.2122941Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2123043Z # tritonclient 2025-10-10T01:33:18.2123115Z pytorch-lightning==2.5.5 2025-10-10T01:33:18.2123193Z  # via 2025-10-10T01:33:18.2123252Z # lightly 2025-10-10T01:33:18.2123343Z # lightning 2025-10-10T01:33:18.2123430Z pytrec-eval-terrier==0.5.7 2025-10-10T01:33:18.2123509Z  # via 2025-10-10T01:33:18.2123586Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2123671Z # mteb 2025-10-10T01:33:18.2123730Z pytz==2024.2 2025-10-10T01:33:18.2123807Z  # via 2025-10-10T01:33:18.2123886Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2123942Z # pandas 2025-10-10T01:33:18.2124030Z # typepy 2025-10-10T01:33:18.2124090Z pyyaml==6.0.2 2025-10-10T01:33:18.2124166Z  # via 2025-10-10T01:33:18.2124241Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2124300Z # accelerate 2025-10-10T01:33:18.2124378Z # albumentations 2025-10-10T01:33:18.2124451Z # datamodel-code-generator 2025-10-10T01:33:18.2124506Z # datasets 2025-10-10T01:33:18.2124570Z # genai-perf 2025-10-10T01:33:18.2124633Z # huggingface-hub 2025-10-10T01:33:18.2124695Z # jsonargparse 2025-10-10T01:33:18.2124857Z # lightning 2025-10-10T01:33:18.2124919Z # mlflow-skinny 2025-10-10T01:33:18.2124981Z # omegaconf 2025-10-10T01:33:18.2125042Z # peft 2025-10-10T01:33:18.2125108Z # pytorch-lightning 2025-10-10T01:33:18.2125168Z # ray 2025-10-10T01:33:18.2125226Z # responses 2025-10-10T01:33:18.2125285Z # schemathesis 2025-10-10T01:33:18.2125354Z # timm 2025-10-10T01:33:18.2125415Z # transformers 2025-10-10T01:33:18.2125507Z # vocos 2025-10-10T01:33:18.2125568Z rapidfuzz==3.12.1 2025-10-10T01:33:18.2125646Z  # via 2025-10-10T01:33:18.2125722Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2125802Z # jiwer 2025-10-10T01:33:18.2125866Z rasterio==1.4.3 2025-10-10T01:33:18.2125943Z  # via 2025-10-10T01:33:18.2126012Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2126071Z # rioxarray 2025-10-10T01:33:18.2126129Z # terratorch 2025-10-10T01:33:18.2126213Z # torchgeo 2025-10-10T01:33:18.2126279Z ray==2.48.0 2025-10-10T01:33:18.2126356Z  # via 2025-10-10T01:33:18.2126428Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2126609Z # -r requirements/test.in 2025-10-10T01:33:18.2126677Z redis==5.2.0 2025-10-10T01:33:18.2126762Z  # via 2025-10-10T01:33:18.2126833Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2126927Z # tensorizer 2025-10-10T01:33:18.2126990Z referencing==0.35.1 2025-10-10T01:33:18.2127066Z  # via 2025-10-10T01:33:18.2127138Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2127197Z # jsonschema 2025-10-10T01:33:18.2127340Z # jsonschema-specifications 2025-10-10T01:33:18.2127454Z regex==2024.9.11 2025-10-10T01:33:18.2127606Z  # via 2025-10-10T01:33:18.2127725Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2127783Z # nltk 2025-10-10T01:33:18.2127850Z # open-clip-torch 2025-10-10T01:33:18.2127913Z # sacrebleu 2025-10-10T01:33:18.2127968Z # tiktoken 2025-10-10T01:33:18.2128087Z # transformers 2025-10-10T01:33:18.2128148Z requests==2.32.3 2025-10-10T01:33:18.2128225Z  # via 2025-10-10T01:33:18.2128308Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2128382Z # buildkite-test-collector 2025-10-10T01:33:18.2128443Z # databricks-sdk 2025-10-10T01:33:18.2128504Z # datasets 2025-10-10T01:33:18.2128559Z # docker 2025-10-10T01:33:18.2128618Z # evaluate 2025-10-10T01:33:18.2128691Z # google-api-core 2025-10-10T01:33:18.2128761Z # google-cloud-storage 2025-10-10T01:33:18.2128831Z # huggingface-hub 2025-10-10T01:33:18.2128891Z # lightly 2025-10-10T01:33:18.2128952Z # lm-eval 2025-10-10T01:33:18.2129015Z # mistral-common 2025-10-10T01:33:18.2129076Z # mlflow-skinny 2025-10-10T01:33:18.2129136Z # mteb 2025-10-10T01:33:18.2129192Z # pooch 2025-10-10T01:33:18.2129247Z # ray 2025-10-10T01:33:18.2129311Z # responses 2025-10-10T01:33:18.2129371Z # schemathesis 2025-10-10T01:33:18.2129449Z # starlette-testclient 2025-10-10T01:33:18.2129509Z # tiktoken 2025-10-10T01:33:18.2129605Z # transformers 2025-10-10T01:33:18.2129678Z responses==0.25.3 2025-10-10T01:33:18.2129755Z  # via 2025-10-10T01:33:18.2129826Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2129921Z # genai-perf 2025-10-10T01:33:18.2129987Z rfc3339-validator==0.1.4 2025-10-10T01:33:18.2130070Z  # via 2025-10-10T01:33:18.2130140Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2130228Z # jsonschema 2025-10-10T01:33:18.2130292Z rfc3987==1.3.8 2025-10-10T01:33:18.2130369Z  # via 2025-10-10T01:33:18.2130445Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2130533Z # jsonschema 2025-10-10T01:33:18.2130590Z rich==13.9.4 2025-10-10T01:33:18.2130674Z  # via 2025-10-10T01:33:18.2130742Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2130800Z # genai-perf 2025-10-10T01:33:18.2130863Z # lightning 2025-10-10T01:33:18.2131018Z # mteb 2025-10-10T01:33:18.2131112Z # typer 2025-10-10T01:33:18.2131175Z rioxarray==0.19.0 2025-10-10T01:33:18.2131261Z  # via 2025-10-10T01:33:18.2131341Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2131431Z # terratorch 2025-10-10T01:33:18.2131502Z rouge-score==0.1.2 2025-10-10T01:33:18.2131580Z  # via 2025-10-10T01:33:18.2131649Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2131739Z # lm-eval 2025-10-10T01:33:18.2131800Z rpds-py==0.20.1 2025-10-10T01:33:18.2131875Z  # via 2025-10-10T01:33:18.2131950Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2132011Z # jsonschema 2025-10-10T01:33:18.2132107Z # referencing 2025-10-10T01:33:18.2132163Z rsa==4.9.1 2025-10-10T01:33:18.2132240Z  # via 2025-10-10T01:33:18.2132313Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2132402Z # google-auth 2025-10-10T01:33:18.2132463Z rtree==1.4.0 2025-10-10T01:33:18.2132556Z  # via 2025-10-10T01:33:18.2132626Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2132717Z # torchgeo 2025-10-10T01:33:18.2132865Z runai-model-streamer==0.14.0 2025-10-10T01:33:18.2132950Z  # via 2025-10-10T01:33:18.2133027Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2133142Z # -r requirements/test.in 2025-10-10T01:33:18.2133224Z runai-model-streamer-gcs==0.14.0 2025-10-10T01:33:18.2133302Z  # via 2025-10-10T01:33:18.2133372Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2133483Z # runai-model-streamer 2025-10-10T01:33:18.2133557Z runai-model-streamer-s3==0.14.0 2025-10-10T01:33:18.2133640Z  # via 2025-10-10T01:33:18.2133709Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2133812Z # runai-model-streamer 2025-10-10T01:33:18.2133877Z s3transfer==0.10.3 2025-10-10T01:33:18.2133956Z  # via 2025-10-10T01:33:18.2134025Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2134116Z # boto3 2025-10-10T01:33:18.2134176Z sacrebleu==2.4.3 2025-10-10T01:33:18.2134257Z  # via 2025-10-10T01:33:18.2134331Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2134427Z # lm-eval 2025-10-10T01:33:18.2134495Z safetensors==0.4.5 2025-10-10T01:33:18.2134573Z  # via 2025-10-10T01:33:18.2134648Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2134705Z # accelerate 2025-10-10T01:33:18.2134770Z # open-clip-torch 2025-10-10T01:33:18.2134842Z # peft 2025-10-10T01:33:18.2134899Z # timm 2025-10-10T01:33:18.2134992Z # transformers 2025-10-10T01:33:18.2135061Z schemathesis==3.39.15 2025-10-10T01:33:18.2135137Z  # via 2025-10-10T01:33:18.2135210Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2135313Z # -r requirements/test.in 2025-10-10T01:33:18.2135396Z scikit-image==0.25.2 2025-10-10T01:33:18.2135549Z  # via 2025-10-10T01:33:18.2135678Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2135848Z # albumentations 2025-10-10T01:33:18.2135915Z scikit-learn==1.5.2 2025-10-10T01:33:18.2136004Z  # via 2025-10-10T01:33:18.2136084Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2136146Z # albumentations 2025-10-10T01:33:18.2136201Z # librosa 2025-10-10T01:33:18.2136260Z # lm-eval 2025-10-10T01:33:18.2136318Z # mlflow 2025-10-10T01:33:18.2136380Z # mteb 2025-10-10T01:33:18.2136488Z # sentence-transformers 2025-10-10T01:33:18.2136548Z scipy==1.13.1 2025-10-10T01:33:18.2136631Z  # via 2025-10-10T01:33:18.2136700Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2136767Z # albumentations 2025-10-10T01:33:18.2136822Z # bm25s 2025-10-10T01:33:18.2136878Z # librosa 2025-10-10T01:33:18.2136942Z # mlflow 2025-10-10T01:33:18.2137012Z # mteb 2025-10-10T01:33:18.2137072Z # scikit-image 2025-10-10T01:33:18.2137131Z # scikit-learn 2025-10-10T01:33:18.2137207Z # sentence-transformers 2025-10-10T01:33:18.2137397Z # statsmodels 2025-10-10T01:33:18.2137485Z # vocos 2025-10-10T01:33:18.2137572Z segmentation-models-pytorch==0.4.0 2025-10-10T01:33:18.2137654Z  # via 2025-10-10T01:33:18.2137717Z # terratorch 2025-10-10T01:33:18.2137803Z # torchgeo 2025-10-10T01:33:18.2137876Z sentence-transformers==3.2.1 2025-10-10T01:33:18.2137964Z  # via 2025-10-10T01:33:18.2138038Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2138110Z # -r requirements/test.in 2025-10-10T01:33:18.2138191Z # mteb 2025-10-10T01:33:18.2138256Z sentencepiece==0.2.0 2025-10-10T01:33:18.2138349Z  # via 2025-10-10T01:33:18.2138422Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2138520Z # mistral-common 2025-10-10T01:33:18.2138587Z setuptools==77.0.3 2025-10-10T01:33:18.2138665Z  # via 2025-10-10T01:33:18.2138740Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2138807Z # lightning-utilities 2025-10-10T01:33:18.2138874Z # pytablewriter 2025-10-10T01:33:18.2138962Z # torch 2025-10-10T01:33:18.2139021Z shapely==2.1.1 2025-10-10T01:33:18.2139183Z  # via 2025-10-10T01:33:18.2139268Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2139326Z # geopandas 2025-10-10T01:33:18.2139420Z # torchgeo 2025-10-10T01:33:18.2139480Z shellingham==1.5.4 2025-10-10T01:33:18.2139559Z  # via 2025-10-10T01:33:18.2139644Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2139727Z # typer 2025-10-10T01:33:18.2139787Z six==1.16.0 2025-10-10T01:33:18.2139864Z  # via 2025-10-10T01:33:18.2139932Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2139997Z # junit-xml 2025-10-10T01:33:18.2140050Z # lightly 2025-10-10T01:33:18.2140119Z # opencensus 2025-10-10T01:33:18.2140187Z # python-dateutil 2025-10-10T01:33:18.2140250Z # rfc3339-validator 2025-10-10T01:33:18.2140312Z # rouge-score 2025-10-10T01:33:18.2140436Z # segmentation-models-pytorch 2025-10-10T01:33:18.2140504Z smart-open==7.1.0 2025-10-10T01:33:18.2140593Z  # via 2025-10-10T01:33:18.2140668Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2140753Z # ray 2025-10-10T01:33:18.2140819Z smmap==5.0.2 2025-10-10T01:33:18.2140896Z  # via 2025-10-10T01:33:18.2140969Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2141051Z # gitdb 2025-10-10T01:33:18.2141114Z sniffio==1.3.1 2025-10-10T01:33:18.2141190Z  # via 2025-10-10T01:33:18.2141259Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2141317Z # anyio 2025-10-10T01:33:18.2141400Z # httpx 2025-10-10T01:33:18.2141465Z sortedcontainers==2.4.0 2025-10-10T01:33:18.2141548Z  # via 2025-10-10T01:33:18.2141618Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2141709Z # hypothesis 2025-10-10T01:33:18.2141769Z soundfile==0.12.1 2025-10-10T01:33:18.2141845Z  # via 2025-10-10T01:33:18.2141917Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2141990Z # -r requirements/test.in 2025-10-10T01:33:18.2142050Z # librosa 2025-10-10T01:33:18.2142147Z # mistral-common 2025-10-10T01:33:18.2142208Z soxr==0.5.0.post1 2025-10-10T01:33:18.2142290Z  # via 2025-10-10T01:33:18.2142358Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2142414Z # librosa 2025-10-10T01:33:18.2142513Z # mistral-common 2025-10-10T01:33:18.2142573Z sqlalchemy==2.0.41 2025-10-10T01:33:18.2142654Z  # via 2025-10-10T01:33:18.2142733Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2142789Z # alembic 2025-10-10T01:33:18.2142879Z # mlflow 2025-10-10T01:33:18.2142941Z sqlitedict==2.1.0 2025-10-10T01:33:18.2143022Z  # via 2025-10-10T01:33:18.2143091Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2143175Z # lm-eval 2025-10-10T01:33:18.2143244Z sqlparse==0.5.3 2025-10-10T01:33:18.2143321Z  # via 2025-10-10T01:33:18.2143390Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2143581Z # mlflow-skinny 2025-10-10T01:33:18.2143650Z starlette==0.46.2 2025-10-10T01:33:18.2143738Z  # via 2025-10-10T01:33:18.2143811Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2143868Z # fastapi 2025-10-10T01:33:18.2143933Z # schemathesis 2025-10-10T01:33:18.2144040Z # starlette-testclient 2025-10-10T01:33:18.2144119Z starlette-testclient==0.4.1 2025-10-10T01:33:18.2144197Z  # via 2025-10-10T01:33:18.2144267Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2144365Z # schemathesis 2025-10-10T01:33:18.2144427Z statsmodels==0.14.4 2025-10-10T01:33:18.2144506Z  # via 2025-10-10T01:33:18.2144581Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2144671Z # genai-perf 2025-10-10T01:33:18.2144733Z sympy==1.13.3 2025-10-10T01:33:18.2144810Z  # via 2025-10-10T01:33:18.2144877Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2144939Z # einx 2025-10-10T01:33:18.2145024Z # torch 2025-10-10T01:33:18.2145086Z tabledata==1.3.3 2025-10-10T01:33:18.2145170Z  # via 2025-10-10T01:33:18.2145317Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2145425Z # pytablewriter 2025-10-10T01:33:18.2145485Z tabulate==0.9.0 2025-10-10T01:33:18.2145562Z  # via 2025-10-10T01:33:18.2145641Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2145726Z # sacrebleu 2025-10-10T01:33:18.2145788Z tblib==3.1.0 2025-10-10T01:33:18.2145873Z  # via 2025-10-10T01:33:18.2145942Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2146055Z # -r requirements/test.in 2025-10-10T01:33:18.2146114Z tcolorpy==0.1.6 2025-10-10T01:33:18.2146192Z  # via 2025-10-10T01:33:18.2146266Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2146360Z # pytablewriter 2025-10-10T01:33:18.2146422Z tenacity==9.0.0 2025-10-10T01:33:18.2146498Z  # via 2025-10-10T01:33:18.2146565Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2146629Z # lm-eval 2025-10-10T01:33:18.2146712Z # plotly 2025-10-10T01:33:18.2146777Z tensorboardx==2.6.4 2025-10-10T01:33:18.2146862Z  # via 2025-10-10T01:33:18.2146937Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2147027Z # lightning 2025-10-10T01:33:18.2147087Z tensorizer==2.10.1 2025-10-10T01:33:18.2147164Z  # via 2025-10-10T01:33:18.2147241Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2147345Z # -r requirements/test.in 2025-10-10T01:33:18.2147606Z terratorch @ git+https://github.com/IBM/terratorch.git@07184fcf91a1324f831ff521dd238d97fe350e3e 2025-10-10T01:33:18.2147730Z  # via -r requirements/test.in 2025-10-10T01:33:18.2147797Z threadpoolctl==3.5.0 2025-10-10T01:33:18.2147879Z  # via 2025-10-10T01:33:18.2147951Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2148048Z # scikit-learn 2025-10-10T01:33:18.2148108Z tifffile==2025.3.30 2025-10-10T01:33:18.2148184Z  # via 2025-10-10T01:33:18.2148264Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2148323Z # scikit-image 2025-10-10T01:33:18.2148418Z # terratorch 2025-10-10T01:33:18.2148489Z tiktoken==0.7.0 2025-10-10T01:33:18.2148569Z  # via 2025-10-10T01:33:18.2148644Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2148699Z # lm-eval 2025-10-10T01:33:18.2148802Z # mistral-common 2025-10-10T01:33:18.2148866Z timm==1.0.17 2025-10-10T01:33:18.2148948Z  # via 2025-10-10T01:33:18.2149019Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2149086Z # -r requirements/test.in 2025-10-10T01:33:18.2149153Z # open-clip-torch 2025-10-10T01:33:18.2149230Z # segmentation-models-pytorch 2025-10-10T01:33:18.2149287Z # terratorch 2025-10-10T01:33:18.2149378Z # torchgeo 2025-10-10T01:33:18.2149437Z tokenizers==0.22.0 2025-10-10T01:33:18.2149515Z  # via 2025-10-10T01:33:18.2149583Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2149739Z # -r requirements/test.in 2025-10-10T01:33:18.2149838Z # transformers 2025-10-10T01:33:18.2149894Z tomli==2.2.1 2025-10-10T01:33:18.2149979Z  # via 2025-10-10T01:33:18.2150054Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2150146Z # schemathesis 2025-10-10T01:33:18.2150208Z tomli-w==1.2.0 2025-10-10T01:33:18.2150284Z  # via 2025-10-10T01:33:18.2150353Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2150448Z # schemathesis 2025-10-10T01:33:18.2150916Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.10.0a0%2Bgit344e636-cp312-cp312-linux_x86_64.whl#sha256=61b81d5619a3dff1b95456c948bf99d131288c4916a1b2d71da5834655529eee 2025-10-10T01:33:18.2150999Z  # via 2025-10-10T01:33:18.2151067Z # -r requirements/test.in 2025-10-10T01:33:18.2151127Z # accelerate 2025-10-10T01:33:18.2151189Z # bitsandbytes 2025-10-10T01:33:18.2151259Z # efficientnet-pytorch 2025-10-10T01:33:18.2151316Z # encodec 2025-10-10T01:33:18.2151385Z # fastsafetensors 2025-10-10T01:33:18.2151438Z # kornia 2025-10-10T01:33:18.2151498Z # lightly 2025-10-10T01:33:18.2151699Z # lightning 2025-10-10T01:33:18.2151814Z # lm-eval 2025-10-10T01:33:18.2151926Z # mteb 2025-10-10T01:33:18.2152044Z # open-clip-torch 2025-10-10T01:33:18.2152149Z # peft 2025-10-10T01:33:18.2152251Z # pretrainedmodels 2025-10-10T01:33:18.2152320Z # pytorch-lightning 2025-10-10T01:33:18.2152393Z # runai-model-streamer 2025-10-10T01:33:18.2152470Z # segmentation-models-pytorch 2025-10-10T01:33:18.2152544Z # sentence-transformers 2025-10-10T01:33:18.2152600Z # tensorizer 2025-10-10T01:33:18.2152663Z # terratorch 2025-10-10T01:33:18.2152720Z # timm 2025-10-10T01:33:18.2152777Z # torchaudio 2025-10-10T01:33:18.2152831Z # torchgeo 2025-10-10T01:33:18.2152892Z # torchmetrics 2025-10-10T01:33:18.2152949Z # torchvision 2025-10-10T01:33:18.2153032Z # vector-quantize-pytorch 2025-10-10T01:33:18.2153142Z # vocos 2025-10-10T01:33:18.2153665Z torchaudio @ file:///var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0%2B87ff22e-cp312-cp312-linux_x86_64.whl#sha256=2eac501979e64f67c113a81572de033ce6ec78d775f6402cacf95aa651581299 2025-10-10T01:33:18.2153753Z  # via 2025-10-10T01:33:18.2153822Z # -r requirements/test.in 2025-10-10T01:33:18.2153885Z # encodec 2025-10-10T01:33:18.2153968Z # vocos 2025-10-10T01:33:18.2154029Z torchgeo==0.7.0 2025-10-10T01:33:18.2154135Z  # via terratorch 2025-10-10T01:33:18.2154248Z torchmetrics==1.8.2 2025-10-10T01:33:18.2154403Z  # via 2025-10-10T01:33:18.2154511Z # lightning 2025-10-10T01:33:18.2154631Z # pytorch-lightning 2025-10-10T01:33:18.2154741Z # terratorch 2025-10-10T01:33:18.2154865Z # torchgeo 2025-10-10T01:33:18.2155398Z torchvision @ file:///var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0%2B966da7e-cp312-cp312-linux_x86_64.whl#sha256=79aa422fd81e11b575186dd520ce1848b0356118875ba3f51c957fae9e3f437f 2025-10-10T01:33:18.2155497Z  # via 2025-10-10T01:33:18.2155572Z # -r requirements/test.in 2025-10-10T01:33:18.2155633Z # lightly 2025-10-10T01:33:18.2155701Z # open-clip-torch 2025-10-10T01:33:18.2155767Z # pretrainedmodels 2025-10-10T01:33:18.2155859Z # segmentation-models-pytorch 2025-10-10T01:33:18.2155918Z # terratorch 2025-10-10T01:33:18.2155980Z # timm 2025-10-10T01:33:18.2156070Z # torchgeo 2025-10-10T01:33:18.2156129Z tqdm==4.66.6 2025-10-10T01:33:18.2156222Z  # via 2025-10-10T01:33:18.2156297Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2156362Z # datasets 2025-10-10T01:33:18.2156420Z # evaluate 2025-10-10T01:33:18.2156486Z # huggingface-hub 2025-10-10T01:33:18.2156547Z # lightly 2025-10-10T01:33:18.2156605Z # lightning 2025-10-10T01:33:18.2156663Z # lm-eval 2025-10-10T01:33:18.2156726Z # mteb 2025-10-10T01:33:18.2156887Z # nltk 2025-10-10T01:33:18.2156957Z # open-clip-torch 2025-10-10T01:33:18.2157015Z # peft 2025-10-10T01:33:18.2157078Z # pqdm 2025-10-10T01:33:18.2157155Z # pretrainedmodels 2025-10-10T01:33:18.2157222Z # pytorch-lightning 2025-10-10T01:33:18.2157298Z # segmentation-models-pytorch 2025-10-10T01:33:18.2157376Z # sentence-transformers 2025-10-10T01:33:18.2157441Z # tqdm-multiprocess 2025-10-10T01:33:18.2157552Z # transformers 2025-10-10T01:33:18.2157625Z tqdm-multiprocess==0.0.11 2025-10-10T01:33:18.2157707Z  # via 2025-10-10T01:33:18.2157787Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2157873Z # lm-eval 2025-10-10T01:33:18.2157944Z transformers==4.56.2 2025-10-10T01:33:18.2158024Z  # via 2025-10-10T01:33:18.2158096Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2158177Z # -r requirements/test.in 2025-10-10T01:33:18.2158238Z # genai-perf 2025-10-10T01:33:18.2158302Z # lm-eval 2025-10-10T01:33:18.2158366Z # peft 2025-10-10T01:33:18.2158437Z # sentence-transformers 2025-10-10T01:33:18.2158670Z # transformers-stream-generator 2025-10-10T01:33:18.2158763Z transformers-stream-generator==0.0.5 2025-10-10T01:33:18.2158846Z  # via 2025-10-10T01:33:18.2158933Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2159039Z # -r requirements/test.in 2025-10-10T01:33:18.2159209Z tritonclient==2.51.0 2025-10-10T01:33:18.2159293Z  # via 2025-10-10T01:33:18.2159373Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2159448Z # -r requirements/test.in 2025-10-10T01:33:18.2159541Z # genai-perf 2025-10-10T01:33:18.2159606Z typepy==1.3.2 2025-10-10T01:33:18.2159685Z  # via 2025-10-10T01:33:18.2159753Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2159819Z # dataproperty 2025-10-10T01:33:18.2159880Z # pytablewriter 2025-10-10T01:33:18.2160002Z # tabledata 2025-10-10T01:33:18.2160126Z typer==0.15.2 2025-10-10T01:33:18.2160283Z  # via 2025-10-10T01:33:18.2160424Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2160563Z # fastsafetensors 2025-10-10T01:33:18.2160649Z types-python-dateutil==2.9.0.20241206 2025-10-10T01:33:18.2160736Z  # via 2025-10-10T01:33:18.2160810Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2160897Z # arrow 2025-10-10T01:33:18.2160965Z typeshed-client==2.8.2 2025-10-10T01:33:18.2161042Z  # via 2025-10-10T01:33:18.2161117Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2161212Z # jsonargparse 2025-10-10T01:33:18.2161286Z typing-extensions==4.12.2 2025-10-10T01:33:18.2161370Z  # via 2025-10-10T01:33:18.2161441Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2161507Z # albumentations 2025-10-10T01:33:18.2161565Z # alembic 2025-10-10T01:33:18.2161621Z # fastapi 2025-10-10T01:33:18.2161685Z # graphene 2025-10-10T01:33:18.2161748Z # huggingface-hub 2025-10-10T01:33:18.2161818Z # librosa 2025-10-10T01:33:18.2161875Z # lightning 2025-10-10T01:33:18.2161949Z # lightning-utilities 2025-10-10T01:33:18.2162017Z # mistral-common 2025-10-10T01:33:18.2162079Z # mlflow-skinny 2025-10-10T01:33:18.2162134Z # mteb 2025-10-10T01:33:18.2162206Z # opentelemetry-api 2025-10-10T01:33:18.2162273Z # opentelemetry-sdk 2025-10-10T01:33:18.2162368Z # opentelemetry-semantic-conventions 2025-10-10T01:33:18.2162425Z # pqdm 2025-10-10T01:33:18.2162483Z # pydantic 2025-10-10T01:33:18.2162560Z # pydantic-core 2025-10-10T01:33:18.2162670Z # pydantic-extra-types 2025-10-10T01:33:18.2162799Z # pytorch-lightning 2025-10-10T01:33:18.2162905Z # sqlalchemy 2025-10-10T01:33:18.2163006Z # torch 2025-10-10T01:33:18.2163119Z # torchgeo 2025-10-10T01:33:18.2163211Z # typer 2025-10-10T01:33:18.2163279Z # typeshed-client 2025-10-10T01:33:18.2163408Z # typing-inspection 2025-10-10T01:33:18.2163592Z typing-inspection==0.4.1 2025-10-10T01:33:18.2163695Z  # via 2025-10-10T01:33:18.2163778Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2163869Z # pydantic 2025-10-10T01:33:18.2163936Z tzdata==2024.2 2025-10-10T01:33:18.2164015Z  # via 2025-10-10T01:33:18.2164087Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2164181Z # pandas 2025-10-10T01:33:18.2164245Z uri-template==1.3.0 2025-10-10T01:33:18.2164329Z  # via 2025-10-10T01:33:18.2164400Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2164492Z # jsonschema 2025-10-10T01:33:18.2164559Z urllib3==2.2.3 2025-10-10T01:33:18.2164637Z  # via 2025-10-10T01:33:18.2164713Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2164770Z # blobfile 2025-10-10T01:33:18.2164828Z # botocore 2025-10-10T01:33:18.2164890Z # docker 2025-10-10T01:33:18.2164946Z # lightly 2025-10-10T01:33:18.2165004Z # requests 2025-10-10T01:33:18.2165083Z # responses 2025-10-10T01:33:18.2165184Z # tritonclient 2025-10-10T01:33:18.2165254Z uvicorn==0.35.0 2025-10-10T01:33:18.2165457Z  # via 2025-10-10T01:33:18.2165543Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2165657Z # mlflow-skinny 2025-10-10T01:33:18.2165739Z vector-quantize-pytorch==1.23.2 2025-10-10T01:33:18.2165871Z  # via -r requirements/test.in 2025-10-10T01:33:18.2165936Z virtualenv==20.31.2 2025-10-10T01:33:18.2166019Z  # via 2025-10-10T01:33:18.2166102Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2166183Z # ray 2025-10-10T01:33:18.2166243Z vocos==0.1.0 2025-10-10T01:33:18.2166329Z  # via 2025-10-10T01:33:18.2166400Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2166515Z # -r requirements/test.in 2025-10-10T01:33:18.2166578Z wcwidth==0.2.13 2025-10-10T01:33:18.2166656Z  # via 2025-10-10T01:33:18.2166736Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2166824Z # ftfy 2025-10-10T01:33:18.2166893Z webcolors==24.11.1 2025-10-10T01:33:18.2166973Z  # via 2025-10-10T01:33:18.2167087Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2167268Z # jsonschema 2025-10-10T01:33:18.2167362Z werkzeug==3.1.3 2025-10-10T01:33:18.2167454Z  # via 2025-10-10T01:33:18.2167535Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2167595Z # flask 2025-10-10T01:33:18.2167695Z # schemathesis 2025-10-10T01:33:18.2167756Z word2number==1.1 2025-10-10T01:33:18.2167834Z  # via 2025-10-10T01:33:18.2167934Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2168225Z # lm-eval 2025-10-10T01:33:18.2168299Z wrapt==1.17.2 2025-10-10T01:33:18.2168388Z  # via 2025-10-10T01:33:18.2168526Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2168632Z # smart-open 2025-10-10T01:33:18.2168695Z xarray==2025.7.1 2025-10-10T01:33:18.2168773Z  # via 2025-10-10T01:33:18.2168854Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2168958Z # rioxarray 2025-10-10T01:33:18.2169024Z xxhash==3.5.0 2025-10-10T01:33:18.2169107Z  # via 2025-10-10T01:33:18.2169179Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2169242Z # datasets 2025-10-10T01:33:18.2169330Z # evaluate 2025-10-10T01:33:18.2169399Z yarl==1.17.1 2025-10-10T01:33:18.2169483Z  # via 2025-10-10T01:33:18.2169555Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2169616Z # aiohttp 2025-10-10T01:33:18.2169710Z # schemathesis 2025-10-10T01:33:18.2169770Z zipp==3.23.0 2025-10-10T01:33:18.2169854Z  # via 2025-10-10T01:33:18.2169926Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2170037Z # importlib-metadata 2025-10-10T01:33:18.2170098Z zstandard==0.23.0 2025-10-10T01:33:18.2170178Z  # via 2025-10-10T01:33:18.2170263Z # -c snapshot_constraint.txt 2025-10-10T01:33:18.2170350Z # lm-eval 2025-10-10T01:33:18.3394074Z 2025-10-10 01:33:18,338 [INFO] cli.lib.common.pip_helper: pip installing packages: /opt/conda/envs/py_3.12/bin/python -m uv pip install -r test.txt 2025-10-10T01:33:18.3394468Z 2025-10-10 01:33:18,339 [INFO] cli.lib.common.utils: [cmd] /opt/conda/envs/py_3.12/bin/python -m uv pip install -r test.txt 2025-10-10T01:33:18.3871386Z Using Python 3.12.11 environment at: /opt/conda/envs/py_3.12 2025-10-10T01:33:18.5247405Z Resolved 324 packages in 132ms 2025-10-10T01:33:18.5519756Z Updating https://github.com/IBM/terratorch.git (07184fcf91a1324f831ff521dd238d97fe350e3e) 2025-10-10T01:33:18.5520232Z Updating https://github.com/EleutherAI/lm-evaluation-harness.git (206b7722158f58c35b7ffcd53b035fdbdda5126d) 2025-10-10T01:33:18.6679555Z Downloading shapely (3.0MiB) 2025-10-10T01:33:18.6682614Z Downloading zstandard (5.2MiB) 2025-10-10T01:33:18.6686269Z Downloading numpy (17.1MiB) 2025-10-10T01:33:18.6689941Z Downloading kaleido (76.2MiB) 2025-10-10T01:33:18.6693103Z Downloading pydantic-core (1.9MiB) 2025-10-10T01:33:18.6830928Z Downloading hf-xet (3.0MiB) 2025-10-10T01:33:18.6832544Z Downloading pillow (4.3MiB) 2025-10-10T01:33:18.6835961Z Downloading pyogrio (26.4MiB) 2025-10-10T01:33:18.6840344Z Downloading rasterio (21.2MiB) 2025-10-10T01:33:18.6842110Z Downloading statsmodels (10.2MiB) 2025-10-10T01:33:18.6843972Z Downloading sqlalchemy (3.2MiB) 2025-10-10T01:33:18.6847293Z Downloading runai-model-streamer-gcs (4.2MiB) 2025-10-10T01:33:18.6848276Z Downloading black (1.7MiB) 2025-10-10T01:33:18.6851124Z Downloading h5py (4.7MiB) 2025-10-10T01:33:18.6853463Z Downloading networkx (1.6MiB) 2025-10-10T01:33:18.6856496Z Downloading mlflow-skinny (6.0MiB) 2025-10-10T01:33:18.6997622Z Downloading pycryptodomex (2.2MiB) 2025-10-10T01:33:18.6999639Z Downloading mistral-common (6.2MiB) 2025-10-10T01:33:18.7002865Z Downloading py-spy (2.6MiB) 2025-10-10T01:33:18.7003830Z Downloading pyarrow (38.2MiB) 2025-10-10T01:33:18.7006259Z Downloading grpcio (5.6MiB) 2025-10-10T01:33:18.7008681Z Downloading transformers (11.1MiB) 2025-10-10T01:33:18.7011367Z Downloading decord (13.0MiB) 2025-10-10T01:33:18.7014744Z Downloading polars (33.2MiB) 2025-10-10T01:33:18.7017846Z Downloading opencv-python-headless (47.7MiB) 2025-10-10T01:33:18.7019432Z Downloading virtualenv (5.8MiB) 2025-10-10T01:33:18.7022144Z Downloading ray (66.9MiB) 2025-10-10T01:33:18.7025230Z Downloading rapidfuzz (3.0MiB) 2025-10-10T01:33:18.7027544Z Downloading pyproj (9.1MiB) 2025-10-10T01:33:18.7029849Z Downloading tokenizers (3.2MiB) 2025-10-10T01:33:18.7032428Z Downloading matplotlib (7.9MiB) 2025-10-10T01:33:18.7034562Z Downloading bitsandbytes (69.5MiB) 2025-10-10T01:33:18.7094034Z Downloading python-box (4.1MiB) 2025-10-10T01:33:18.7096841Z Downloading scipy (36.4MiB) 2025-10-10T01:33:18.7099951Z Downloading mteb (1.5MiB) 2025-10-10T01:33:18.7102647Z Downloading fonttools (4.7MiB) 2025-10-10T01:33:18.7136485Z Downloading mlflow (27.7MiB) 2025-10-10T01:33:18.7139342Z Downloading tritonclient (13.3MiB) 2025-10-10T01:33:18.7142636Z Downloading fiona (16.4MiB) 2025-10-10T01:33:18.7145374Z Downloading cramjam (2.2MiB) 2025-10-10T01:33:18.7148100Z Downloading timm (2.4MiB) 2025-10-10T01:33:18.7151067Z Downloading scikit-image (14.3MiB) 2025-10-10T01:33:18.7154085Z Downloading plotly (18.2MiB) 2025-10-10T01:33:18.7156917Z Downloading pandas (12.1MiB) 2025-10-10T01:33:18.7160245Z Downloading scikit-learn (12.3MiB) 2025-10-10T01:33:18.7163247Z Downloading botocore (12.1MiB) 2025-10-10T01:33:18.7167059Z Downloading python-rapidjson (1.6MiB) 2025-10-10T01:33:18.7170468Z Downloading runai-model-streamer-s3 (5.6MiB) 2025-10-10T01:33:18.7222930Z Downloading fastparquet (1.7MiB) 2025-10-10T01:33:18.7226331Z Downloading kornia-rs (2.6MiB) 2025-10-10T01:33:19.6528443Z Downloading python-rapidjson 2025-10-10T01:33:19.6745869Z Downloading fastparquet 2025-10-10T01:33:19.6991738Z Downloading open-clip-torch (1.5MiB) 2025-10-10T01:33:19.6993387Z Downloading nltk (1.4MiB) 2025-10-10T01:33:19.7343864Z Downloading pydantic-core 2025-10-10T01:33:19.7656577Z Downloading kiwisolver (1.4MiB) 2025-10-10T01:33:19.8402413Z Downloading black 2025-10-10T01:33:19.8574576Z Downloading fastsafetensors (1.4MiB) 2025-10-10T01:33:19.9217209Z Downloading cramjam 2025-10-10T01:33:19.9453060Z Downloading xarray (1.3MiB) 2025-10-10T01:33:19.9829609Z Downloading networkx 2025-10-10T01:33:20.0051573Z Downloading aiohttp (1.2MiB) 2025-10-10T01:33:20.0061139Z Downloading pycryptodomex 2025-10-10T01:33:20.0111167Z Downloading py-spy 2025-10-10T01:33:20.0276638Z Downloading setuptools (1.2MiB) 2025-10-10T01:33:20.0277466Z Downloading sentencepiece (1.2MiB) 2025-10-10T01:33:20.0827239Z Downloading kornia-rs 2025-10-10T01:33:20.0973290Z Downloading pygments (1.1MiB) 2025-10-10T01:33:20.1753971Z Downloading timm 2025-10-10T01:33:20.1970764Z Downloading soundfile (1.1MiB) 2025-10-10T01:33:20.2362914Z Downloading hf-xet 2025-10-10T01:33:20.2461446Z Downloading rapidfuzz 2025-10-10T01:33:20.2526523Z Downloading tiktoken (1.0MiB) 2025-10-10T01:33:20.2707064Z Downloading kornia (1.0MiB) 2025-10-10T01:33:20.2957752Z Downloading tokenizers 2025-10-10T01:33:20.3224061Z Downloading sqlalchemy 2025-10-10T01:33:20.3386543Z Downloading shapely 2025-10-10T01:33:20.3544814Z Updated https://github.com/IBM/terratorch.git (07184fcf91a1324f831ff521dd238d97fe350e3e) 2025-10-10T01:33:20.3550733Z Building terratorch @ git+https://github.com/IBM/terratorch.git@07184fcf91a1324f831ff521dd238d97fe350e3e 2025-10-10T01:33:20.4033457Z Downloading open-clip-torch 2025-10-10T01:33:20.4213739Z Downloading nltk 2025-10-10T01:33:20.4408047Z Downloading mteb 2025-10-10T01:33:20.5061349Z Downloading kiwisolver 2025-10-10T01:33:20.5831100Z Downloading fastsafetensors 2025-10-10T01:33:20.6413384Z Downloading sentencepiece 2025-10-10T01:33:20.6625181Z Downloading xarray 2025-10-10T01:33:20.6862481Z Downloading aiohttp 2025-10-10T01:33:20.7973489Z Downloading python-box 2025-10-10T01:33:20.8295367Z Downloading soundfile 2025-10-10T01:33:20.8535777Z Downloading tiktoken 2025-10-10T01:33:20.8549648Z Downloading pygments 2025-10-10T01:33:20.8691172Z Downloading runai-model-streamer-gcs 2025-10-10T01:33:20.8852852Z Downloading pillow 2025-10-10T01:33:20.9498957Z Downloading setuptools 2025-10-10T01:33:21.0345538Z Downloading fonttools 2025-10-10T01:33:21.0587311Z Downloading h5py 2025-10-10T01:33:21.1379615Z Updated https://github.com/EleutherAI/lm-evaluation-harness.git (206b7722158f58c35b7ffcd53b035fdbdda5126d) 2025-10-10T01:33:21.1385981Z Building lm-eval @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@206b7722158f58c35b7ffcd53b035fdbdda5126d 2025-10-10T01:33:21.1933942Z Built terratorch @ git+https://github.com/IBM/terratorch.git@07184fcf91a1324f831ff521dd238d97fe350e3e 2025-10-10T01:33:21.2221544Z Downloading kornia 2025-10-10T01:33:21.2586331Z Downloading zstandard 2025-10-10T01:33:21.4787169Z Downloading runai-model-streamer-s3 2025-10-10T01:33:21.5420901Z Downloading virtualenv 2025-10-10T01:33:21.5727410Z Downloading grpcio 2025-10-10T01:33:21.8523486Z Downloading mistral-common 2025-10-10T01:33:21.9862781Z Building rouge-score==0.1.2 2025-10-10T01:33:22.0593103Z Downloading mlflow-skinny 2025-10-10T01:33:22.0836595Z Building encodec==0.1.1 2025-10-10T01:33:22.0851752Z Building pretrainedmodels==0.7.4 2025-10-10T01:33:22.0852944Z Building antlr4-python3-runtime==4.9.3 2025-10-10T01:33:22.0888951Z Building docopt==0.6.2 2025-10-10T01:33:22.0890157Z Building sqlitedict==2.1.0 2025-10-10T01:33:22.0923188Z Building transformers-stream-generator==0.0.5 2025-10-10T01:33:22.0925801Z Building efficientnet-pytorch==0.7.1 2025-10-10T01:33:22.0926459Z Building word2number==1.1 2025-10-10T01:33:22.5492168Z Built rouge-score==0.1.2 2025-10-10T01:33:22.6576430Z Built docopt==0.6.2 2025-10-10T01:33:22.6741109Z Downloading matplotlib 2025-10-10T01:33:22.6751944Z Downloading pyproj 2025-10-10T01:33:22.6766096Z Built sqlitedict==2.1.0 2025-10-10T01:33:22.6768066Z Built efficientnet-pytorch==0.7.1 2025-10-10T01:33:22.7631954Z Built encodec==0.1.1 2025-10-10T01:33:22.7783623Z Built pretrainedmodels==0.7.4 2025-10-10T01:33:22.8093250Z Built antlr4-python3-runtime==4.9.3 2025-10-10T01:33:22.8419438Z Built word2number==1.1 2025-10-10T01:33:22.8498784Z Built transformers-stream-generator==0.0.5 2025-10-10T01:33:22.9877775Z Downloading statsmodels 2025-10-10T01:33:23.1716185Z Downloading pandas 2025-10-10T01:33:23.3418013Z Downloading decord 2025-10-10T01:33:23.3697319Z Downloading scikit-learn 2025-10-10T01:33:23.4387318Z Downloading tritonclient 2025-10-10T01:33:23.4462610Z Downloading botocore 2025-10-10T01:33:23.4606352Z Downloading transformers 2025-10-10T01:33:23.5859250Z Downloading scikit-image 2025-10-10T01:33:23.7051368Z Downloading fiona 2025-10-10T01:33:23.7255928Z Downloading numpy 2025-10-10T01:33:24.0809040Z Downloading rasterio 2025-10-10T01:33:24.3681976Z Downloading pyogrio 2025-10-10T01:33:24.7579311Z Downloading polars 2025-10-10T01:33:24.9405787Z Downloading mlflow 2025-10-10T01:33:25.1696023Z Downloading pyarrow 2025-10-10T01:33:25.1772459Z Downloading scipy 2025-10-10T01:33:25.6009448Z Downloading opencv-python-headless 2025-10-10T01:33:25.9525131Z Downloading bitsandbytes 2025-10-10T01:33:26.2279873Z Downloading kaleido 2025-10-10T01:33:26.2635628Z Downloading ray 2025-10-10T01:33:27.5274231Z Downloading plotly 2025-10-10T01:33:35.7790166Z Built lm-eval @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@206b7722158f58c35b7ffcd53b035fdbdda5126d 2025-10-10T01:33:36.0180116Z Prepared 289 packages in 17.48s 2025-10-10T01:33:38.7918737Z Uninstalled 98 packages in 2.77s 2025-10-10T01:33:40.0045054Z Installed 289 packages in 1.21s 2025-10-10T01:33:40.0056919Z - absl-py==2.3.1 2025-10-10T01:33:40.0057798Z + absl-py==2.1.0 2025-10-10T01:33:40.0058476Z + accelerate==1.0.1 2025-10-10T01:33:40.0059542Z + aenum==3.1.16 2025-10-10T01:33:40.0060669Z + affine==2.4.0 2025-10-10T01:33:40.0061836Z - aiohappyeyeballs==2.6.1 2025-10-10T01:33:40.0062824Z + aiohappyeyeballs==2.4.3 2025-10-10T01:33:40.0063988Z - aiohttp==3.13.0 2025-10-10T01:33:40.0064667Z + aiohttp==3.10.11 2025-10-10T01:33:40.0065682Z + aiohttp-cors==0.8.1 2025-10-10T01:33:40.0066104Z - aiosignal==1.4.0 2025-10-10T01:33:40.0068034Z + aiosignal==1.3.1 2025-10-10T01:33:40.0068746Z + albucore==0.0.16 2025-10-10T01:33:40.0069858Z + albumentations==1.4.6 2025-10-10T01:33:40.0071106Z + alembic==1.16.4 2025-10-10T01:33:40.0072455Z + antlr4-python3-runtime==4.9.3 2025-10-10T01:33:40.0073008Z - anyio==4.11.0 2025-10-10T01:33:40.0073719Z + anyio==4.6.2.post1 2025-10-10T01:33:40.0074568Z + argcomplete==3.5.1 2025-10-10T01:33:40.0074938Z + arrow==1.3.0 2025-10-10T01:33:40.0075644Z - attrs==25.4.0 2025-10-10T01:33:40.0077054Z + attrs==24.2.0 2025-10-10T01:33:40.0077576Z + backoff==2.2.1 2025-10-10T01:33:40.0078549Z + bitsandbytes==0.46.1 2025-10-10T01:33:40.0079647Z + black==24.10.0 2025-10-10T01:33:40.0081015Z + blinker==1.9.0 2025-10-10T01:33:40.0081793Z + blobfile==3.0.0 2025-10-10T01:33:40.0082774Z + bm25s==0.2.13 2025-10-10T01:33:40.0083391Z - boto3==1.35.42 2025-10-10T01:33:40.0083913Z + boto3==1.35.57 2025-10-10T01:33:40.0085026Z - botocore==1.35.99 2025-10-10T01:33:40.0086039Z + botocore==1.35.57 2025-10-10T01:33:40.0086707Z + bounded-pool-executor==0.0.3 2025-10-10T01:33:40.0087953Z + buildkite-test-collector==0.1.9 2025-10-10T01:33:40.0089382Z - cachetools==6.2.0 2025-10-10T01:33:40.0090087Z + cachetools==5.5.2 2025-10-10T01:33:40.0090708Z - certifi==2025.10.5 2025-10-10T01:33:40.0091415Z + certifi==2024.8.30 2025-10-10T01:33:40.0092819Z - cffi==2.0.0 2025-10-10T01:33:40.0093482Z + cffi==1.17.1 2025-10-10T01:33:40.0094784Z + chardet==5.2.0 2025-10-10T01:33:40.0095348Z - charset-normalizer==3.4.3 2025-10-10T01:33:40.0096371Z + charset-normalizer==3.4.0 2025-10-10T01:33:40.0097226Z - click==8.3.0 2025-10-10T01:33:40.0098313Z + click==8.1.7 2025-10-10T01:33:40.0099167Z + click-plugins==1.1.1.2 2025-10-10T01:33:40.0099928Z + cligj==0.7.2 2025-10-10T01:33:40.0101661Z + colorful==0.5.6 2025-10-10T01:33:40.0101981Z + contourpy==1.3.0 2025-10-10T01:33:40.0102678Z + coverage==7.10.6 2025-10-10T01:33:40.0103622Z + cramjam==2.9.0 2025-10-10T01:33:40.0104895Z + cycler==0.12.1 2025-10-10T01:33:40.0105648Z + databricks-sdk==0.59.0 2025-10-10T01:33:40.0106892Z + datamodel-code-generator==0.26.3 2025-10-10T01:33:40.0107941Z + dataproperty==1.0.1 2025-10-10T01:33:40.0108888Z + datasets==3.0.2 2025-10-10T01:33:40.0109874Z - decorator==5.2.1 2025-10-10T01:33:40.0110685Z + decorator==5.1.1 2025-10-10T01:33:40.0112464Z + decord==0.6.0 2025-10-10T01:33:40.0112887Z - dill==0.3.7 2025-10-10T01:33:40.0113368Z + dill==0.3.8 2025-10-10T01:33:40.0114098Z + distlib==0.3.9 2025-10-10T01:33:40.0116157Z - dnspython==2.8.0 2025-10-10T01:33:40.0116730Z + dnspython==2.7.0 2025-10-10T01:33:40.0117258Z + docopt==0.6.2 2025-10-10T01:33:40.0118868Z + docstring-parser==0.17.0 2025-10-10T01:33:40.0119747Z + efficientnet-pytorch==0.7.1 2025-10-10T01:33:40.0121400Z + einx==0.3.0 2025-10-10T01:33:40.0122504Z - email-validator==2.3.0 2025-10-10T01:33:40.0123405Z + email-validator==2.2.0 2025-10-10T01:33:40.0125085Z + encodec==0.1.1 2025-10-10T01:33:40.0125572Z + eval-type-backport==0.2.2 2025-10-10T01:33:40.0127093Z + evaluate==0.4.3 2025-10-10T01:33:40.0127764Z - fastapi==0.118.2 2025-10-10T01:33:40.0128724Z + fastapi==0.116.1 2025-10-10T01:33:40.0130339Z + fastparquet==2024.11.0 2025-10-10T01:33:40.0131697Z - fastrlock==0.8.3 2025-10-10T01:33:40.0132394Z + fastrlock==0.8.2 2025-10-10T01:33:40.0133605Z + fastsafetensors==0.1.10 2025-10-10T01:33:40.0134512Z - filelock==3.18.0 2025-10-10T01:33:40.0147872Z + filelock==3.16.1 2025-10-10T01:33:40.0148400Z + fiona==1.10.1 2025-10-10T01:33:40.0148867Z + flask==3.1.1 2025-10-10T01:33:40.0149359Z + fonttools==4.55.0 2025-10-10T01:33:40.0149781Z + fqdn==1.5.1 2025-10-10T01:33:40.0150162Z - frozenlist==1.8.0 2025-10-10T01:33:40.0150452Z + frozenlist==1.5.0 2025-10-10T01:33:40.0150741Z - fsspec==2025.9.0 2025-10-10T01:33:40.0151116Z + fsspec==2024.9.0 2025-10-10T01:33:40.0151422Z + ftfy==6.3.1 2025-10-10T01:33:40.0151743Z + genai-perf==0.0.8 2025-10-10T01:33:40.0152261Z + genson==1.3.0 2025-10-10T01:33:40.0152568Z + geopandas==1.0.1 2025-10-10T01:33:40.0152863Z - gitpython==3.1.45 2025-10-10T01:33:40.0153552Z + gitpython==3.1.44 2025-10-10T01:33:40.0153877Z + google-api-core==2.24.2 2025-10-10T01:33:40.0154326Z - google-auth==2.41.1 2025-10-10T01:33:40.0154720Z + google-auth==2.40.2 2025-10-10T01:33:40.0155108Z + google-cloud-core==2.4.3 2025-10-10T01:33:40.0155560Z + google-cloud-storage==3.4.0 2025-10-10T01:33:40.0155930Z + google-crc32c==1.7.1 2025-10-10T01:33:40.0156484Z + google-resumable-media==2.7.2 2025-10-10T01:33:40.0157152Z + googleapis-common-protos==1.70.0 2025-10-10T01:33:40.0157921Z + graphene==3.4.3 2025-10-10T01:33:40.0159274Z + graphql-core==3.2.6 2025-10-10T01:33:40.0160409Z + graphql-relay==3.2.0 2025-10-10T01:33:40.0161370Z + greenlet==3.2.3 2025-10-10T01:33:40.0162107Z - grpcio==1.75.1 2025-10-10T01:33:40.0163280Z + grpcio==1.71.0 2025-10-10T01:33:40.0164340Z + gunicorn==23.0.0 2025-10-10T01:33:40.0165300Z - h11==0.16.0 2025-10-10T01:33:40.0166951Z + h11==0.14.0 2025-10-10T01:33:40.0167423Z + h5py==3.13.0 2025-10-10T01:33:40.0168132Z + harfile==0.3.0 2025-10-10T01:33:40.0169067Z - hf-xet==1.1.10 2025-10-10T01:33:40.0170171Z + hf-xet==1.1.7 2025-10-10T01:33:40.0170667Z + hiredis==3.0.0 2025-10-10T01:33:40.0171297Z - httpcore==1.0.9 2025-10-10T01:33:40.0172450Z + httpcore==1.0.6 2025-10-10T01:33:40.0173548Z - httpx==0.28.1 2025-10-10T01:33:40.0175053Z + httpx==0.27.2 2025-10-10T01:33:40.0175632Z - huggingface-hub==0.35.3 2025-10-10T01:33:40.0177297Z + huggingface-hub==0.34.3 2025-10-10T01:33:40.0177830Z + humanize==4.11.0 2025-10-10T01:33:40.0179073Z + hydra-core==1.3.2 2025-10-10T01:33:40.0179586Z - hypothesis==6.56.4 2025-10-10T01:33:40.0180690Z + hypothesis==6.131.0 2025-10-10T01:33:40.0182014Z + hypothesis-graphql==0.11.1 2025-10-10T01:33:40.0182573Z + hypothesis-jsonschema==0.23.1 2025-10-10T01:33:40.0183487Z + importlib-metadata==8.7.0 2025-10-10T01:33:40.0184514Z + importlib-resources==6.5.2 2025-10-10T01:33:40.0185069Z + inflect==5.6.2 2025-10-10T01:33:40.0186273Z - iniconfig==2.1.0 2025-10-10T01:33:40.0187267Z + iniconfig==2.0.0 2025-10-10T01:33:40.0188609Z + isoduration==20.11.0 2025-10-10T01:33:40.0189348Z + isort==5.13.2 2025-10-10T01:33:40.0190110Z + itsdangerous==2.2.0 2025-10-10T01:33:40.0191261Z + jiwer==3.0.5 2025-10-10T01:33:40.0192347Z - joblib==1.5.2 2025-10-10T01:33:40.0193467Z + joblib==1.4.2 2025-10-10T01:33:40.0194912Z + jsonargparse==4.35.0 2025-10-10T01:33:40.0195343Z + jsonlines==4.0.0 2025-10-10T01:33:40.0196842Z + jsonpointer==3.0.0 2025-10-10T01:33:40.0197477Z - jsonschema==4.25.1 2025-10-10T01:33:40.0198067Z + jsonschema==4.23.0 2025-10-10T01:33:40.0199600Z - jsonschema-specifications==2025.9.1 2025-10-10T01:33:40.0200369Z + jsonschema-specifications==2024.10.1 2025-10-10T01:33:40.0201182Z + junit-xml==1.9 2025-10-10T01:33:40.0202078Z + kaleido==0.2.1 2025-10-10T01:33:40.0203716Z + kiwisolver==1.4.7 2025-10-10T01:33:40.0205380Z + kornia==0.8.1 2025-10-10T01:33:40.0205900Z + kornia-rs==0.1.9 2025-10-10T01:33:40.0206271Z + libnacl==2.1.0 2025-10-10T01:33:40.0206714Z - librosa==0.10.2 2025-10-10T01:33:40.0208126Z + librosa==0.10.2.post1 2025-10-10T01:33:40.0209231Z + lightly==1.5.20 2025-10-10T01:33:40.0209929Z + lightly-utils==0.0.2 2025-10-10T01:33:40.0211049Z + lightning==2.5.1.post0 2025-10-10T01:33:40.0211702Z + lightning-utilities==0.14.3 2025-10-10T01:33:40.0213769Z + lm-eval==0.4.9.1 (from git+https://github.com/EleutherAI/lm-evaluation-harness.git@206b7722158f58c35b7ffcd53b035fdbdda5126d) 2025-10-10T01:33:40.0214601Z + mako==1.3.10 2025-10-10T01:33:40.0215126Z - markdown==3.9 2025-10-10T01:33:40.0215769Z + markdown==3.8.2 2025-10-10T01:33:40.0216551Z - markdown-it-py==4.0.0 2025-10-10T01:33:40.0217905Z + markdown-it-py==3.0.0 2025-10-10T01:33:40.0219042Z - markupsafe==3.0.3 2025-10-10T01:33:40.0219629Z + markupsafe==3.0.1 2025-10-10T01:33:40.0220868Z + matplotlib==3.9.2 2025-10-10T01:33:40.0221745Z + mbstrdecoder==1.1.3 2025-10-10T01:33:40.0222864Z - mistral-common==1.8.5 2025-10-10T01:33:40.0224159Z + mistral-common==1.8.2 2025-10-10T01:33:40.0225011Z + mlflow==2.22.0 2025-10-10T01:33:40.0226081Z + mlflow-skinny==2.22.0 2025-10-10T01:33:40.0226540Z + more-itertools==10.5.0 2025-10-10T01:33:40.0227407Z - msgpack==1.1.2 2025-10-10T01:33:40.0229098Z + msgpack==1.1.0 2025-10-10T01:33:40.0229681Z + mteb==1.38.11 2025-10-10T01:33:40.0230602Z - multidict==6.7.0 2025-10-10T01:33:40.0231357Z + multidict==6.1.0 2025-10-10T01:33:40.0231948Z + multiprocess==0.70.16 2025-10-10T01:33:40.0233139Z + munch==4.0.0 2025-10-10T01:33:40.0234085Z - mypy-extensions==1.1.0 2025-10-10T01:33:40.0235262Z + mypy-extensions==1.0.0 2025-10-10T01:33:40.0235794Z - networkx==2.8.8 2025-10-10T01:33:40.0236921Z + networkx==3.2.1 2025-10-10T01:33:40.0237948Z + nltk==3.9.1 2025-10-10T01:33:40.0239603Z + num2words==0.5.14 2025-10-10T01:33:40.0240247Z + numexpr==2.10.1 2025-10-10T01:33:40.0241343Z - numpy==2.2.6 2025-10-10T01:33:40.0242264Z + numpy==1.26.4 2025-10-10T01:33:40.0243294Z + omegaconf==2.3.0 2025-10-10T01:33:40.0244477Z + open-clip-torch==2.32.0 2025-10-10T01:33:40.0245531Z + opencensus==0.11.4 2025-10-10T01:33:40.0246452Z + opencensus-context==0.1.3 2025-10-10T01:33:40.0247093Z - opencv-python-headless==4.12.0.88 2025-10-10T01:33:40.0248635Z + opencv-python-headless==4.11.0.86 2025-10-10T01:33:40.0249666Z + opentelemetry-api==1.35.0 2025-10-10T01:33:40.0250877Z + opentelemetry-exporter-prometheus==0.56b0 2025-10-10T01:33:40.0251800Z + opentelemetry-proto==1.36.0 2025-10-10T01:33:40.0252970Z + opentelemetry-sdk==1.35.0 2025-10-10T01:33:40.0253932Z + opentelemetry-semantic-conventions==0.56b0 2025-10-10T01:33:40.0254727Z - packaging==25.0 2025-10-10T01:33:40.0255659Z + packaging==24.2 2025-10-10T01:33:40.0256356Z - pandas==2.0.3 2025-10-10T01:33:40.0257962Z + pandas==2.2.3 2025-10-10T01:33:40.0258519Z + pathvalidate==3.2.1 2025-10-10T01:33:40.0259622Z + patsy==1.0.1 2025-10-10T01:33:40.0260313Z + peft==0.16.0 2025-10-10T01:33:40.0261056Z - pillow==11.0.0 2025-10-10T01:33:40.0261903Z + pillow==10.4.0 2025-10-10T01:33:40.0263207Z - platformdirs==4.5.0 2025-10-10T01:33:40.0264345Z + platformdirs==4.3.6 2025-10-10T01:33:40.0265174Z + plotly==5.24.1 2025-10-10T01:33:40.0266220Z - pluggy==1.6.0 2025-10-10T01:33:40.0266812Z + pluggy==1.5.0 2025-10-10T01:33:40.0268006Z + polars==1.29.0 2025-10-10T01:33:40.0269051Z + portalocker==2.10.1 2025-10-10T01:33:40.0270301Z + pqdm==0.2.0 2025-10-10T01:33:40.0271268Z + pretrainedmodels==0.7.4 2025-10-10T01:33:40.0272308Z - prometheus-client==0.23.1 2025-10-10T01:33:40.0273091Z + prometheus-client==0.22.0 2025-10-10T01:33:40.0274162Z - propcache==0.3.2 2025-10-10T01:33:40.0274718Z + propcache==0.2.0 2025-10-10T01:33:40.0275281Z + proto-plus==1.26.1 2025-10-10T01:33:40.0276450Z - protobuf==5.29.5 2025-10-10T01:33:40.0277160Z + protobuf==5.28.3 2025-10-10T01:33:40.0277744Z - psutil==7.1.0 2025-10-10T01:33:40.0278687Z + psutil==6.1.0 2025-10-10T01:33:40.0280042Z + py==1.11.0 2025-10-10T01:33:40.0280706Z + py-spy==0.4.0 2025-10-10T01:33:40.0281244Z + pyarrow==18.0.0 2025-10-10T01:33:40.0282910Z + pybind11==2.13.6 2025-10-10T01:33:40.0283592Z + pycocotools==2.0.8 2025-10-10T01:33:40.0284132Z - pycparser==2.23 2025-10-10T01:33:40.0284636Z + pycparser==2.22 2025-10-10T01:33:40.0286141Z + pycryptodomex==3.22.0 2025-10-10T01:33:40.0286677Z - pydantic==2.12.0 2025-10-10T01:33:40.0287158Z + pydantic==2.11.7 2025-10-10T01:33:40.0288628Z - pydantic-core==2.41.1 2025-10-10T01:33:40.0289402Z + pydantic-core==2.33.2 2025-10-10T01:33:40.0290639Z - pydantic-extra-types==2.10.6 2025-10-10T01:33:40.0291213Z + pydantic-extra-types==2.10.5 2025-10-10T01:33:40.0291749Z - pygments==2.15.0 2025-10-10T01:33:40.0292870Z + pygments==2.18.0 2025-10-10T01:33:40.0293779Z + pyogrio==0.11.0 2025-10-10T01:33:40.0294563Z + pyparsing==3.2.0 2025-10-10T01:33:40.0295706Z + pyproj==3.7.1 2025-10-10T01:33:40.0296695Z + pyrate-limiter==3.7.0 2025-10-10T01:33:40.0297187Z + pystemmer==3.0.0 2025-10-10T01:33:40.0298335Z + pytablewriter==1.2.0 2025-10-10T01:33:40.0298902Z - pytest==7.3.2 2025-10-10T01:33:40.0300101Z + pytest==8.3.5 2025-10-10T01:33:40.0300794Z + pytest-asyncio==0.24.0 2025-10-10T01:33:40.0301301Z + pytest-cov==6.3.0 2025-10-10T01:33:40.0302483Z + pytest-forked==1.6.0 2025-10-10T01:33:40.0303436Z + pytest-mock==3.14.0 2025-10-10T01:33:40.0305143Z + pytest-shard==0.1.2 2025-10-10T01:33:40.0305672Z - pytest-subtests==0.13.1 2025-10-10T01:33:40.0306192Z + pytest-subtests==0.14.1 2025-10-10T01:33:40.0306708Z + pytest-timeout==2.3.1 2025-10-10T01:33:40.0307240Z + python-box==7.3.2 2025-10-10T01:33:40.0307979Z + python-rapidjson==1.20 2025-10-10T01:33:40.0308818Z + pytorch-lightning==2.5.5 2025-10-10T01:33:40.0309527Z + pytrec-eval-terrier==0.5.7 2025-10-10T01:33:40.0310640Z - pytz==2025.2 2025-10-10T01:33:40.0311152Z + pytz==2024.2 2025-10-10T01:33:40.0312554Z + rapidfuzz==3.12.1 2025-10-10T01:33:40.0313635Z + rasterio==1.4.3 2025-10-10T01:33:40.0314225Z - ray==2.49.2 2025-10-10T01:33:40.0315191Z + ray==2.48.0 2025-10-10T01:33:40.0316631Z - redis==6.4.0 2025-10-10T01:33:40.0317423Z + redis==5.2.0 2025-10-10T01:33:40.0318666Z - referencing==0.36.2 2025-10-10T01:33:40.0319292Z + referencing==0.35.1 2025-10-10T01:33:40.0320249Z - regex==2025.9.18 2025-10-10T01:33:40.0321600Z + regex==2024.9.11 2025-10-10T01:33:40.0322668Z - requests==2.32.5 2025-10-10T01:33:40.0323269Z + requests==2.32.3 2025-10-10T01:33:40.0324345Z + responses==0.25.3 2025-10-10T01:33:40.0325342Z + rfc3339-validator==0.1.4 2025-10-10T01:33:40.0326899Z + rfc3987==1.3.8 2025-10-10T01:33:40.0327677Z - rich==14.2.0 2025-10-10T01:33:40.0329075Z + rich==13.9.4 2025-10-10T01:33:40.0330052Z + rioxarray==0.19.0 2025-10-10T01:33:40.0330828Z + rouge-score==0.1.2 2025-10-10T01:33:40.0332504Z - rpds-py==0.27.1 2025-10-10T01:33:40.0332813Z + rpds-py==0.20.1 2025-10-10T01:33:40.0333473Z + rtree==1.4.0 2025-10-10T01:33:40.0334732Z + runai-model-streamer==0.14.0 2025-10-10T01:33:40.0335986Z + runai-model-streamer-gcs==0.14.0 2025-10-10T01:33:40.0336657Z + runai-model-streamer-s3==0.14.0 2025-10-10T01:33:40.0337775Z - s3transfer==0.10.4 2025-10-10T01:33:40.0338608Z + s3transfer==0.10.3 2025-10-10T01:33:40.0339433Z + sacrebleu==2.4.3 2025-10-10T01:33:40.0340204Z - safetensors==0.6.2 2025-10-10T01:33:40.0341008Z + safetensors==0.4.5 2025-10-10T01:33:40.0342032Z + schemathesis==3.39.15 2025-10-10T01:33:40.0343174Z - scikit-image==0.22.0 2025-10-10T01:33:40.0343669Z + scikit-image==0.25.2 2025-10-10T01:33:40.0344704Z - scikit-learn==1.7.2 2025-10-10T01:33:40.0345215Z + scikit-learn==1.5.2 2025-10-10T01:33:40.0345756Z - scipy==1.14.1 2025-10-10T01:33:40.0347146Z + scipy==1.13.1 2025-10-10T01:33:40.0347782Z + segmentation-models-pytorch==0.4.0 2025-10-10T01:33:40.0349138Z + sentence-transformers==3.2.1 2025-10-10T01:33:40.0350061Z - sentencepiece==0.2.1 2025-10-10T01:33:40.0351853Z + sentencepiece==0.2.0 2025-10-10T01:33:40.0352386Z - setuptools==78.1.1 2025-10-10T01:33:40.0352950Z + setuptools==77.0.3 2025-10-10T01:33:40.0353757Z + shapely==2.1.1 2025-10-10T01:33:40.0354977Z - six==1.17.0 2025-10-10T01:33:40.0356290Z + six==1.16.0 2025-10-10T01:33:40.0357119Z + smart-open==7.1.0 2025-10-10T01:33:40.0358249Z - soundfile==0.13.1 2025-10-10T01:33:40.0359506Z + soundfile==0.12.1 2025-10-10T01:33:40.0360497Z - soxr==1.0.0 2025-10-10T01:33:40.0362120Z + soxr==0.5.0.post1 2025-10-10T01:33:40.0363024Z + sqlalchemy==2.0.41 2025-10-10T01:33:40.0364358Z + sqlitedict==2.1.0 2025-10-10T01:33:40.0365418Z + sqlparse==0.5.3 2025-10-10T01:33:40.0366212Z - starlette==0.48.0 2025-10-10T01:33:40.0366790Z + starlette==0.46.2 2025-10-10T01:33:40.0367752Z + starlette-testclient==0.4.1 2025-10-10T01:33:40.0368289Z + statsmodels==0.14.4 2025-10-10T01:33:40.0369130Z + tabledata==1.3.3 2025-10-10T01:33:40.0370210Z + tblib==3.1.0 2025-10-10T01:33:40.0370969Z + tcolorpy==0.1.6 2025-10-10T01:33:40.0371722Z + tenacity==9.0.0 2025-10-10T01:33:40.0372730Z + tensorboardx==2.6.4 2025-10-10T01:33:40.0373275Z + tensorizer==2.10.1 2025-10-10T01:33:40.0375102Z + terratorch==1.0.2 (from git+https://github.com/IBM/terratorch.git@07184fcf91a1324f831ff521dd238d97fe350e3e) 2025-10-10T01:33:40.0376061Z - threadpoolctl==3.6.0 2025-10-10T01:33:40.0376628Z + threadpoolctl==3.5.0 2025-10-10T01:33:40.0377154Z - tifffile==2025.10.4 2025-10-10T01:33:40.0377766Z + tifffile==2025.3.30 2025-10-10T01:33:40.0378194Z - tiktoken==0.12.0 2025-10-10T01:33:40.0379233Z + tiktoken==0.7.0 2025-10-10T01:33:40.0380229Z + timm==1.0.17 2025-10-10T01:33:40.0380671Z - tokenizers==0.22.1 2025-10-10T01:33:40.0381363Z + tokenizers==0.22.0 2025-10-10T01:33:40.0382194Z + tomli==2.2.1 2025-10-10T01:33:40.0383792Z + tomli-w==1.2.0 2025-10-10T01:33:40.0385292Z - torch==2.10.0a0+git344e636 (from file:///var/lib/jenkins/workspace/dist/torch-2.10.0a0%2Bgit344e636-cp312-cp312-linux_x86_64.whl) 2025-10-10T01:33:40.0387223Z + torch==2.10.0a0+git344e636 (from file:///var/lib/jenkins/workspace/dist/torch-2.10.0a0+git344e636-cp312-cp312-linux_x86_64.whl#sha256=61b81d5619a3dff1b95456c948bf99d131288c4916a1b2d71da5834655529eee) 2025-10-10T01:33:40.0388802Z - torchaudio==2.8.0a0+87ff22e (from file:///var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0%2B87ff22e-cp312-cp312-linux_x86_64.whl) 2025-10-10T01:33:40.0391011Z + torchaudio==2.8.0a0+87ff22e (from file:///var/lib/jenkins/workspace/dist/audio/torchaudio-2.8.0a0+87ff22e-cp312-cp312-linux_x86_64.whl#sha256=2eac501979e64f67c113a81572de033ce6ec78d775f6402cacf95aa651581299) 2025-10-10T01:33:40.0392507Z + torchgeo==0.7.0 2025-10-10T01:33:40.0393039Z + torchmetrics==1.8.2 2025-10-10T01:33:40.0393904Z - torchvision==0.22.0a0+966da7e (from file:///var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0%2B966da7e-cp312-cp312-linux_x86_64.whl) 2025-10-10T01:33:40.0396017Z + torchvision==0.22.0a0+966da7e (from file:///var/lib/jenkins/workspace/dist/vision/torchvision-0.22.0a0+966da7e-cp312-cp312-linux_x86_64.whl#sha256=79aa422fd81e11b575186dd520ce1848b0356118875ba3f51c957fae9e3f437f) 2025-10-10T01:33:40.0397230Z - tqdm==4.67.1 2025-10-10T01:33:40.0397543Z + tqdm==4.66.6 2025-10-10T01:33:40.0397945Z + tqdm-multiprocess==0.0.11 2025-10-10T01:33:40.0398279Z - transformers==4.57.0 2025-10-10T01:33:40.0398815Z + transformers==4.56.2 2025-10-10T01:33:40.0399853Z + transformers-stream-generator==0.0.5 2025-10-10T01:33:40.0400568Z + tritonclient==2.51.0 2025-10-10T01:33:40.0402233Z + typepy==1.3.2 2025-10-10T01:33:40.0402804Z - typer==0.19.2 2025-10-10T01:33:40.0404369Z + typer==0.15.2 2025-10-10T01:33:40.0405528Z + types-python-dateutil==2.9.0.20241206 2025-10-10T01:33:40.0406139Z + typeshed-client==2.8.2 2025-10-10T01:33:40.0406715Z - typing-extensions==4.15.0 2025-10-10T01:33:40.0408239Z + typing-extensions==4.12.2 2025-10-10T01:33:40.0409232Z - typing-inspection==0.4.2 2025-10-10T01:33:40.0409788Z + typing-inspection==0.4.1 2025-10-10T01:33:40.0410430Z - tzdata==2025.2 2025-10-10T01:33:40.0411054Z + tzdata==2024.2 2025-10-10T01:33:40.0411775Z + uri-template==1.3.0 2025-10-10T01:33:40.0412492Z - urllib3==2.5.0 2025-10-10T01:33:40.0414113Z + urllib3==2.2.3 2025-10-10T01:33:40.0414751Z - uvicorn==0.37.0 2025-10-10T01:33:40.0415547Z + uvicorn==0.35.0 2025-10-10T01:33:40.0417361Z + vector-quantize-pytorch==1.23.2 2025-10-10T01:33:40.0417983Z + virtualenv==20.31.2 2025-10-10T01:33:40.0418790Z + vocos==0.1.0 2025-10-10T01:33:40.0419359Z + wcwidth==0.2.13 2025-10-10T01:33:40.0419875Z + webcolors==24.11.1 2025-10-10T01:33:40.0420263Z + word2number==1.1 2025-10-10T01:33:40.0421449Z - wrapt==1.17.3 2025-10-10T01:33:40.0422087Z + wrapt==1.17.2 2025-10-10T01:33:40.0423139Z + xarray==2025.7.1 2025-10-10T01:33:40.0424204Z + xxhash==3.5.0 2025-10-10T01:33:40.0425261Z - yarl==1.22.0 2025-10-10T01:33:40.0426037Z + yarl==1.17.1 2025-10-10T01:33:40.0427374Z + zipp==3.23.0 2025-10-10T01:33:40.0428094Z + zstandard==0.23.0 2025-10-10T01:33:40.2091448Z 2025-10-10 01:33:40,208 [INFO] cli.lib.core.vllm.vllm_test: Done. installed requirements for test dependencies 2025-10-10T01:33:40.2092113Z 2025-10-10 01:33:40,208 [INFO] cli.lib.core.vllm.vllm_test: Double check installed packages 2025-10-10T01:33:40.2136977Z 2025-10-10 01:33:40,213 [INFO] cli.lib.common.pip_helper: torch already exist with version: 2.10.0a0+git344e636 2025-10-10T01:33:40.2140151Z 2025-10-10 01:33:40,213 [INFO] cli.lib.common.pip_helper: xformers already exist with version: 0.0.33+5d4b92a5.d20251010 2025-10-10T01:33:40.2143873Z 2025-10-10 01:33:40,214 [INFO] cli.lib.common.pip_helper: torchvision already exist with version: 0.22.0a0+966da7e 2025-10-10T01:33:40.2147692Z 2025-10-10 01:33:40,214 [INFO] cli.lib.common.pip_helper: torchaudio already exist with version: 2.8.0a0+87ff22e 2025-10-10T01:33:40.2153598Z 2025-10-10 01:33:40,215 [INFO] cli.lib.common.pip_helper: vllm already exist with version: 0.11.0rc2.dev157+g0ad9951c4.d20251010 2025-10-10T01:33:40.2155357Z 2025-10-10 01:33:40,215 [INFO] cli.lib.core.vllm.vllm_test: Done. checked installed packages 2025-10-10T01:33:40.2155971Z 2025-10-10 01:33:40,215 [INFO] cli.lib.core.vllm.lib: run vllm tests..... 2025-10-10T01:33:40.2156466Z 2025-10-10 01:33:40,215 [INFO] cli.lib.core.vllm.lib: Running tests: Basic models test 2025-10-10T01:33:40.2157169Z 2025-10-10 01:33:40,215 [INFO] cli.lib.core.vllm.lib: Running step: pytest -v -s models/test_transformers.py 2025-10-10T01:33:40.2157779Z 2025-10-10 01:33:40,215 [INFO] cli.lib.common.utils: [shell] pytest -v -s models/test_transformers.py 2025-10-10T01:33:50.6108563Z INFO 10-10 01:33:50 [__init__.py:215] Automatically detected platform cuda. 2025-10-10T01:33:53.1213638Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset. 2025-10-10T01:33:53.1216261Z The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session" 2025-10-10T01:33:53.1217926Z 2025-10-10T01:33:53.1218180Z warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) 2025-10-10T01:33:53.1276522Z ============================= test session starts ============================== 2025-10-10T01:33:53.1277091Z platform linux -- Python 3.12.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/conda/envs/py_3.12/bin/python 2025-10-10T01:33:53.1380305Z cachedir: .pytest_cache 2025-10-10T01:33:53.1381217Z hypothesis profile 'ci' -> database=None, deadline=None, print_blob=True, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-10-10T01:33:53.1382136Z rootdir: /var/lib/jenkins/workspace/vllm 2025-10-10T01:33:53.1382441Z configfile: pyproject.toml 2025-10-10T01:33:53.1383444Z plugins: cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, xdist-3.3.1, xdoctest-1.3.0, asyncio-0.24.0, cov-6.3.0, buildkite-test-collector-0.1.9, schemathesis-3.39.15, anyio-4.6.2.post1, hydra-core-1.3.2, mock-3.14.0, subtests-0.14.1, timeout-2.3.1, hypothesis-6.131.0, shard-0.1.2, forked-1.6.0 2025-10-10T01:33:53.1384817Z asyncio: mode=Mode.STRICT, default_loop_scope=None 2025-10-10T01:33:56.3490524Z collecting ... WARNING 10-10 01:33:56 [interface.py:488] Current platform cuda does not have '_pytestfixturefunction' attribute. 2025-10-10T01:33:56.8389232Z WARNING 10-10 01:33:56 [interface.py:488] Current platform cuda does not have '__test__' attribute. 2025-10-10T01:33:56.8390028Z WARNING 10-10 01:33:56 [interface.py:488] Current platform cuda does not have '__bases__' attribute. 2025-10-10T01:33:56.8390711Z WARNING 10-10 01:33:56 [interface.py:488] Current platform cuda does not have '__test__' attribute. 2025-10-10T01:33:56.8391734Z WARNING 10-10 01:33:56 [interface.py:488] Current platform cuda does not have '_schemathesis_test' attribute. 2025-10-10T01:33:56.8419053Z  2025-10-10T01:33:56.8423202Z collecting 12 items  2025-10-10T01:33:56.8424242Z collected 12 items  2025-10-10T01:33:56.8428187Z Running 12 items in this shard: tests/models/test_transformers.py::test_models[meta-llama/Llama-3.2-1B-Instruct-transformers], tests/models/test_transformers.py::test_models[hmellor/Ilama-3.2-1B-auto], tests/models/test_transformers.py::test_models[allenai/OLMoE-1B-7B-0924-transformers], tests/models/test_transformers.py::test_hybrid_attention, tests/models/test_transformers.py::test_distributed, tests/models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ-quantization_kwargs0], tests/models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ-quantization_kwargs1], tests/models/test_transformers.py::test_quantization[5-32-meta-llama/Llama-3.2-1B-Instruct-quantization_kwargs2], tests/models/test_transformers.py::test_embed_loading[Qwen/Qwen3-Embedding-0.6B], tests/models/test_transformers.py::test_embed_loading[meta-llama/Llama-3.2-1B-Instruct], tests/models/test_transformers.py::test_pooling[TransformersEmbeddingModel], tests/models/test_transformers.py::test_pooling[TransformersForSequenceClassification] 2025-10-10T01:33:56.8432718Z 2025-10-10T01:33:56.8681150Z models/test_transformers.py::test_models[meta-llama/Llama-3.2-1B-Instruct-transformers] INFO 10-10 01:33:56 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model_impl': 'transformers', 'model': 'meta-llama/Llama-3.2-1B-Instruct'} 2025-10-10T01:33:56.9562615Z 2025-10-10T01:33:56.9563484Z config.json: 0% 0.00/877 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:05.7506832Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:05.7507492Z 2025-10-10T01:34:05.7508048Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:05.7508568Z 2025-10-10T01:34:05.7509535Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:05.7510395Z 2025-10-10T01:34:05.7510951Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:05.7511502Z 2025-10-10T01:34:05.7512029Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7512537Z 2025-10-10T01:34:05.7513512Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:05.7514361Z 2025-10-10T01:34:05.7515054Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:05.7515706Z 2025-10-10T01:34:05.7516671Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:05.7517551Z 2025-10-10T01:34:05.7518177Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:05.7518789Z 2025-10-10T01:34:05.7519495Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7520043Z 2025-10-10T01:34:05.7521023Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:05.7521917Z 2025-10-10T01:34:05.7522398Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] self._init_executor() 2025-10-10T01:34:05.7522876Z 2025-10-10T01:34:05.7523899Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:05.7524757Z 2025-10-10T01:34:05.7525301Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:05.7525807Z 2025-10-10T01:34:05.7526715Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:05.7527831Z 2025-10-10T01:34:05.7528418Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:05.7528765Z 2025-10-10T01:34:05.7529080Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7529387Z 2025-10-10T01:34:05.7529918Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:05.7530386Z 2025-10-10T01:34:05.7530673Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:05.7530955Z 2025-10-10T01:34:05.7531223Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7531486Z 2025-10-10T01:34:05.7532132Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:05.7532617Z 2025-10-10T01:34:05.7532926Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:05.7533208Z 2025-10-10T01:34:05.7533469Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7533740Z 2025-10-10T01:34:05.7534271Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:05.7534755Z 2025-10-10T01:34:05.7535072Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:05.7535374Z 2025-10-10T01:34:05.7535661Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7535940Z 2025-10-10T01:34:05.7536376Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:05.7536784Z 2025-10-10T01:34:05.7537123Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:05.7537448Z 2025-10-10T01:34:05.7537747Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7538021Z 2025-10-10T01:34:05.7538383Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:05.7538727Z 2025-10-10T01:34:05.7539088Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:05.7539435Z 2025-10-10T01:34:05.7539813Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:05.7540182Z 2025-10-10T01:34:05.7540525Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:05.7540865Z 2025-10-10T01:34:05.7541229Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:05.7541578Z 2025-10-10T01:34:05.7541961Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:05.7542404Z 2025-10-10T01:34:05.7542927Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:05.7543386Z 2025-10-10T01:34:05.7543751Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:05.7544087Z 2025-10-10T01:34:05.7544617Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:05.7545098Z 2025-10-10T01:34:05.7545466Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:05.7545839Z 2025-10-10T01:34:05.7546402Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:05.7547019Z 2025-10-10T01:34:05.7547329Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:05.7547628Z 2025-10-10T01:34:05.7548275Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:05.7548854Z 2025-10-10T01:34:05.7549223Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:05.7549573Z 2025-10-10T01:34:05.7549832Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7550090Z 2025-10-10T01:34:05.7550655Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:05.7551168Z 2025-10-10T01:34:05.7551474Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:05.7551768Z 2025-10-10T01:34:05.7552044Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7552313Z 2025-10-10T01:34:05.7552923Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:05.7553503Z 2025-10-10T01:34:05.7553796Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:05.7554089Z 2025-10-10T01:34:05.7554344Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7554617Z 2025-10-10T01:34:05.7555214Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:05.7555751Z 2025-10-10T01:34:05.7556065Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:05.7556372Z 2025-10-10T01:34:05.7556644Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7556914Z 2025-10-10T01:34:05.7557440Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:05.7558018Z 2025-10-10T01:34:05.7558305Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:05.7558588Z 2025-10-10T01:34:05.7558857Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7559271Z 2025-10-10T01:34:05.7559803Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:05.7560293Z 2025-10-10T01:34:05.7560601Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:05.7560909Z 2025-10-10T01:34:05.7561149Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:05.7561391Z 2025-10-10T01:34:05.7561992Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:05.7562454Z 2025-10-10T01:34:05.7562719Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] raise RuntimeError( 2025-10-10T01:34:05.7562980Z 2025-10-10T01:34:05.7563552Z (EngineCore_DP0 pid=740) ERROR 10-10 01:34:05 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:05.7564248Z (EngineCore_DP0 pid=740) Process EngineCore_DP0: 2025-10-10T01:34:05.7564676Z (EngineCore_DP0 pid=740) Traceback (most recent call last): 2025-10-10T01:34:05.7565272Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:05.7565802Z (EngineCore_DP0 pid=740) self.run() 2025-10-10T01:34:05.7566329Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:05.7566886Z (EngineCore_DP0 pid=740) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:05.7567546Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:05.7568091Z (EngineCore_DP0 pid=740) raise e 2025-10-10T01:34:05.7568681Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:05.7569323Z (EngineCore_DP0 pid=740) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:05.7569771Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7570389Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:05.7571022Z (EngineCore_DP0 pid=740) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:05.7571673Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:05.7572305Z (EngineCore_DP0 pid=740) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:05.7572769Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7573428Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:05.7574007Z (EngineCore_DP0 pid=740) self._init_executor() 2025-10-10T01:34:05.7574665Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:05.7575485Z (EngineCore_DP0 pid=740) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:05.7576219Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:05.7576941Z (EngineCore_DP0 pid=740) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:05.7577432Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7578070Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:05.7578665Z (EngineCore_DP0 pid=740) return func(*args, **kwargs) 2025-10-10T01:34:05.7579059Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7579778Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:05.7580426Z (EngineCore_DP0 pid=740) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:05.7580844Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7581514Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:05.7582169Z (EngineCore_DP0 pid=740) module = importlib.import_module(module_name) 2025-10-10T01:34:05.7582619Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7583191Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:05.7583824Z (EngineCore_DP0 pid=740) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:05.7584309Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7584796Z (EngineCore_DP0 pid=740) File "", line 1387, in _gcd_import 2025-10-10T01:34:05.7585350Z (EngineCore_DP0 pid=740) File "", line 1360, in _find_and_load 2025-10-10T01:34:05.7585936Z (EngineCore_DP0 pid=740) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:05.7586514Z (EngineCore_DP0 pid=740) File "", line 935, in _load_unlocked 2025-10-10T01:34:05.7587083Z (EngineCore_DP0 pid=740) File "", line 999, in exec_module 2025-10-10T01:34:05.7587689Z (EngineCore_DP0 pid=740) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:05.7588434Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:05.7589109Z (EngineCore_DP0 pid=740) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:05.7589821Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:05.7590561Z (EngineCore_DP0 pid=740) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:05.7591337Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:05.7592010Z (EngineCore_DP0 pid=740) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:05.7592907Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:05.7593739Z (EngineCore_DP0 pid=740) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:05.7594220Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7594917Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:05.7595616Z (EngineCore_DP0 pid=740) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:05.7596059Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7597108Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:05.7597959Z (EngineCore_DP0 pid=740) return _is_fa2_supported(device)[0] 2025-10-10T01:34:05.7598420Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7599224Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:05.7599956Z (EngineCore_DP0 pid=740) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:05.7600417Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7601089Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:05.7601725Z (EngineCore_DP0 pid=740) prop = get_device_properties(device) 2025-10-10T01:34:05.7602140Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:05.7602800Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:05.7603455Z (EngineCore_DP0 pid=740) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:05.7603847Z (EngineCore_DP0 pid=740) ^^^^^^^^^^^^ 2025-10-10T01:34:05.7604425Z (EngineCore_DP0 pid=740) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:05.7604982Z (EngineCore_DP0 pid=740) raise RuntimeError( 2025-10-10T01:34:05.7605684Z (EngineCore_DP0 pid=740) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:06.1703890Z FAILED 2025-10-10T01:34:06.1729828Z models/test_transformers.py::test_models[hmellor/Ilama-3.2-1B-auto] INFO 10-10 01:34:06 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model': 'hmellor/Ilama-3.2-1B'} 2025-10-10T01:34:06.3021021Z 2025-10-10T01:34:06.3023141Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:34:06.3023468Z config.json: 1.02kB [00:00, 6.10MB/s] 2025-10-10T01:34:06.3827164Z 2025-10-10T01:34:06.3828800Z configuration_ilama.py: 0% 0.00/94.0 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:08.5750869Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:08.5751284Z 2025-10-10T01:34:08.5751671Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:08.5752052Z 2025-10-10T01:34:08.5752739Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:08.5753375Z 2025-10-10T01:34:08.5754049Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:08.5754448Z 2025-10-10T01:34:08.5754793Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5755137Z 2025-10-10T01:34:08.5755735Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:08.5756284Z 2025-10-10T01:34:08.5756683Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:08.5757077Z 2025-10-10T01:34:08.5757675Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:08.5758141Z 2025-10-10T01:34:08.5758455Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:08.5758755Z 2025-10-10T01:34:08.5759140Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5759422Z 2025-10-10T01:34:08.5760061Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:08.5760652Z 2025-10-10T01:34:08.5761106Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] self._init_executor() 2025-10-10T01:34:08.5761513Z 2025-10-10T01:34:08.5762538Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:08.5763275Z 2025-10-10T01:34:08.5763887Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:08.5764267Z 2025-10-10T01:34:08.5764820Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:08.5765318Z 2025-10-10T01:34:08.5765661Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:08.5765986Z 2025-10-10T01:34:08.5766277Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5766719Z 2025-10-10T01:34:08.5767518Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:08.5768548Z 2025-10-10T01:34:08.5768982Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:08.5769413Z 2025-10-10T01:34:08.5769809Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5770221Z 2025-10-10T01:34:08.5771023Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:08.5771764Z 2025-10-10T01:34:08.5772203Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:08.5772647Z 2025-10-10T01:34:08.5773055Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5773486Z 2025-10-10T01:34:08.5774438Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:08.5775212Z 2025-10-10T01:34:08.5775695Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:08.5776166Z 2025-10-10T01:34:08.5776605Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5777039Z 2025-10-10T01:34:08.5777759Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:08.5778440Z 2025-10-10T01:34:08.5778997Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:08.5779586Z 2025-10-10T01:34:08.5780096Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5780586Z 2025-10-10T01:34:08.5781347Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:08.5781954Z 2025-10-10T01:34:08.5782585Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:08.5783181Z 2025-10-10T01:34:08.5783842Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:08.5784475Z 2025-10-10T01:34:08.5785091Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:08.5785684Z 2025-10-10T01:34:08.5786333Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:08.5786958Z 2025-10-10T01:34:08.5787630Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:08.5788271Z 2025-10-10T01:34:08.5789146Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:08.5789979Z 2025-10-10T01:34:08.5790606Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:08.5791218Z 2025-10-10T01:34:08.5792138Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:08.5793098Z 2025-10-10T01:34:08.5793740Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:08.5794340Z 2025-10-10T01:34:08.5795262Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:08.5796473Z 2025-10-10T01:34:08.5797007Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:08.5797489Z 2025-10-10T01:34:08.5798698Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:08.5799784Z 2025-10-10T01:34:08.5800593Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:08.5801244Z 2025-10-10T01:34:08.5801701Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5802164Z 2025-10-10T01:34:08.5803186Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:08.5804139Z 2025-10-10T01:34:08.5804660Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:08.5805163Z 2025-10-10T01:34:08.5805623Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5806092Z 2025-10-10T01:34:08.5807139Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:08.5808091Z 2025-10-10T01:34:08.5808573Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:08.5809045Z 2025-10-10T01:34:08.5809487Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5809926Z 2025-10-10T01:34:08.5810928Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:08.5811835Z 2025-10-10T01:34:08.5812384Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:08.5812936Z 2025-10-10T01:34:08.5813428Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5813919Z 2025-10-10T01:34:08.5814839Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:08.5815640Z 2025-10-10T01:34:08.5816110Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:08.5816594Z 2025-10-10T01:34:08.5817042Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5817487Z 2025-10-10T01:34:08.5818386Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:08.5819336Z 2025-10-10T01:34:08.5819871Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:08.5820389Z 2025-10-10T01:34:08.5820769Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:08.5821166Z 2025-10-10T01:34:08.5821979Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:08.5822735Z 2025-10-10T01:34:08.5823141Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] raise RuntimeError( 2025-10-10T01:34:08.5823583Z 2025-10-10T01:34:08.5824616Z (EngineCore_DP0 pid=767) ERROR 10-10 01:34:08 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:08.5825867Z (EngineCore_DP0 pid=767) Process EngineCore_DP0: 2025-10-10T01:34:08.5826679Z (EngineCore_DP0 pid=767) Traceback (most recent call last): 2025-10-10T01:34:08.5827700Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:08.5828567Z (EngineCore_DP0 pid=767) self.run() 2025-10-10T01:34:08.5829444Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:08.5830386Z (EngineCore_DP0 pid=767) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:08.5831524Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:08.5832436Z (EngineCore_DP0 pid=767) raise e 2025-10-10T01:34:08.5833428Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:08.5834506Z (EngineCore_DP0 pid=767) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:08.5835274Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5836382Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:08.5837536Z (EngineCore_DP0 pid=767) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:08.5838654Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:08.5839786Z (EngineCore_DP0 pid=767) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:08.5840541Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5841647Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:08.5842621Z (EngineCore_DP0 pid=767) self._init_executor() 2025-10-10T01:34:08.5843714Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:08.5844923Z (EngineCore_DP0 pid=767) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:08.5846205Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:08.5847493Z (EngineCore_DP0 pid=767) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:08.5848360Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5849577Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:08.5850594Z (EngineCore_DP0 pid=767) return func(*args, **kwargs) 2025-10-10T01:34:08.5851264Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5852391Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:08.5853505Z (EngineCore_DP0 pid=767) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:08.5854225Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5855374Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:08.5856508Z (EngineCore_DP0 pid=767) module = importlib.import_module(module_name) 2025-10-10T01:34:08.5857474Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5858511Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:08.5859626Z (EngineCore_DP0 pid=767) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:08.5860498Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5861370Z (EngineCore_DP0 pid=767) File "", line 1387, in _gcd_import 2025-10-10T01:34:08.5862346Z (EngineCore_DP0 pid=767) File "", line 1360, in _find_and_load 2025-10-10T01:34:08.5863387Z (EngineCore_DP0 pid=767) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:08.5864427Z (EngineCore_DP0 pid=767) File "", line 935, in _load_unlocked 2025-10-10T01:34:08.5865422Z (EngineCore_DP0 pid=767) File "", line 999, in exec_module 2025-10-10T01:34:08.5866488Z (EngineCore_DP0 pid=767) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:08.5867839Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:08.5868901Z (EngineCore_DP0 pid=767) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:08.5870049Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:08.5871336Z (EngineCore_DP0 pid=767) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:08.5872745Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:08.5873930Z (EngineCore_DP0 pid=767) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:08.5875360Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:08.5876764Z (EngineCore_DP0 pid=767) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:08.5877602Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5878831Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:08.5880248Z (EngineCore_DP0 pid=767) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:08.5881008Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5882351Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:08.5883594Z (EngineCore_DP0 pid=767) return _is_fa2_supported(device)[0] 2025-10-10T01:34:08.5884230Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5885342Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:08.5886584Z (EngineCore_DP0 pid=767) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:08.5887538Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5889056Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:08.5890243Z (EngineCore_DP0 pid=767) prop = get_device_properties(device) 2025-10-10T01:34:08.5890937Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:08.5892059Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:08.5893206Z (EngineCore_DP0 pid=767) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:08.5893915Z (EngineCore_DP0 pid=767) ^^^^^^^^^^^^ 2025-10-10T01:34:08.5894949Z (EngineCore_DP0 pid=767) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:08.5895958Z (EngineCore_DP0 pid=767) raise RuntimeError( 2025-10-10T01:34:08.5897397Z (EngineCore_DP0 pid=767) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:09.0316927Z FAILED 2025-10-10T01:34:09.3452566Z models/test_transformers.py::test_models[allenai/OLMoE-1B-7B-0924-transformers] SKIPPED 2025-10-10T01:34:09.3481709Z models/test_transformers.py::test_hybrid_attention INFO 10-10 01:34:09 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'max_model_len': 8192, 'block_size': 16, 'disable_log_stats': True, 'enforce_eager': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model_impl': 'transformers', 'model': 'hmellor/tiny-random-Gemma2ForCausalLM'} 2025-10-10T01:34:09.4524832Z 2025-10-10T01:34:09.4525748Z config.json: 0% 0.00/813 [00:00, 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:12.1202184Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:12.1202803Z 2025-10-10T01:34:12.1203411Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:12.1203807Z 2025-10-10T01:34:12.1204500Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:12.1205094Z 2025-10-10T01:34:12.1205491Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:12.1205865Z 2025-10-10T01:34:12.1206208Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1206542Z 2025-10-10T01:34:12.1207155Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:12.1207994Z 2025-10-10T01:34:12.1208581Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:12.1209147Z 2025-10-10T01:34:12.1209787Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:12.1210348Z 2025-10-10T01:34:12.1210738Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:12.1211134Z 2025-10-10T01:34:12.1211428Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1211706Z 2025-10-10T01:34:12.1212219Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:12.1212961Z 2025-10-10T01:34:12.1213519Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] self._init_executor() 2025-10-10T01:34:12.1213964Z 2025-10-10T01:34:12.1214896Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:12.1215840Z 2025-10-10T01:34:12.1216398Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:12.1216755Z 2025-10-10T01:34:12.1217407Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:12.1217925Z 2025-10-10T01:34:12.1218276Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:12.1218621Z 2025-10-10T01:34:12.1218910Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1219195Z 2025-10-10T01:34:12.1219687Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:12.1220163Z 2025-10-10T01:34:12.1220431Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:12.1220697Z 2025-10-10T01:34:12.1220952Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1221205Z 2025-10-10T01:34:12.1221725Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:12.1222202Z 2025-10-10T01:34:12.1222490Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:12.1222777Z 2025-10-10T01:34:12.1223040Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1223345Z 2025-10-10T01:34:12.1223877Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:12.1224358Z 2025-10-10T01:34:12.1224662Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:12.1224956Z 2025-10-10T01:34:12.1225236Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1225617Z 2025-10-10T01:34:12.1226058Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:12.1226475Z 2025-10-10T01:34:12.1226811Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:12.1227144Z 2025-10-10T01:34:12.1227424Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1227711Z 2025-10-10T01:34:12.1228050Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:12.1228385Z 2025-10-10T01:34:12.1228724Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:12.1229062Z 2025-10-10T01:34:12.1229521Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:12.1229884Z 2025-10-10T01:34:12.1230233Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:12.1230564Z 2025-10-10T01:34:12.1230937Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:12.1231288Z 2025-10-10T01:34:12.1231663Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:12.1232030Z 2025-10-10T01:34:12.1232524Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:12.1232993Z 2025-10-10T01:34:12.1233344Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:12.1233681Z 2025-10-10T01:34:12.1234226Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:12.1234710Z 2025-10-10T01:34:12.1235082Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:12.1235448Z 2025-10-10T01:34:12.1236007Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:12.1236513Z 2025-10-10T01:34:12.1236810Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:12.1237102Z 2025-10-10T01:34:12.1237732Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:12.1238299Z 2025-10-10T01:34:12.1238666Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:12.1239011Z 2025-10-10T01:34:12.1239414Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1239682Z 2025-10-10T01:34:12.1240241Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:12.1240844Z 2025-10-10T01:34:12.1241141Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:12.1241438Z 2025-10-10T01:34:12.1241706Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1241981Z 2025-10-10T01:34:12.1242571Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:12.1243116Z 2025-10-10T01:34:12.1243399Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:12.1243678Z 2025-10-10T01:34:12.1243938Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1244203Z 2025-10-10T01:34:12.1244904Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:12.1245436Z 2025-10-10T01:34:12.1245751Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:12.1246064Z 2025-10-10T01:34:12.1246331Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1246607Z 2025-10-10T01:34:12.1247124Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:12.1247628Z 2025-10-10T01:34:12.1247904Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:12.1248197Z 2025-10-10T01:34:12.1248462Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1248738Z 2025-10-10T01:34:12.1249263Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:12.1249739Z 2025-10-10T01:34:12.1250049Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:12.1250352Z 2025-10-10T01:34:12.1250580Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:12.1250817Z 2025-10-10T01:34:12.1251295Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:12.1251748Z 2025-10-10T01:34:12.1251992Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] raise RuntimeError( 2025-10-10T01:34:12.1252259Z 2025-10-10T01:34:12.1252819Z (EngineCore_DP0 pid=773) ERROR 10-10 01:34:12 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:12.1253510Z (EngineCore_DP0 pid=773) Process EngineCore_DP0: 2025-10-10T01:34:12.1253905Z (EngineCore_DP0 pid=773) Traceback (most recent call last): 2025-10-10T01:34:12.1254514Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:12.1255024Z (EngineCore_DP0 pid=773) self.run() 2025-10-10T01:34:12.1255563Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:12.1256210Z (EngineCore_DP0 pid=773) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:12.1256886Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:12.1257435Z (EngineCore_DP0 pid=773) raise e 2025-10-10T01:34:12.1258013Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:12.1258644Z (EngineCore_DP0 pid=773) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:12.1259088Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1259704Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:12.1260338Z (EngineCore_DP0 pid=773) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:12.1261074Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:12.1261697Z (EngineCore_DP0 pid=773) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:12.1262148Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1262798Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:12.1263382Z (EngineCore_DP0 pid=773) self._init_executor() 2025-10-10T01:34:12.1264027Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:12.1264710Z (EngineCore_DP0 pid=773) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:12.1265440Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:12.1266147Z (EngineCore_DP0 pid=773) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:12.1266651Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1267287Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:12.1267862Z (EngineCore_DP0 pid=773) return func(*args, **kwargs) 2025-10-10T01:34:12.1268237Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1268882Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:12.1269515Z (EngineCore_DP0 pid=773) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:12.1269943Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1270605Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:12.1271264Z (EngineCore_DP0 pid=773) module = importlib.import_module(module_name) 2025-10-10T01:34:12.1271701Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1272278Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:12.1272888Z (EngineCore_DP0 pid=773) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:12.1273454Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1273938Z (EngineCore_DP0 pid=773) File "", line 1387, in _gcd_import 2025-10-10T01:34:12.1274495Z (EngineCore_DP0 pid=773) File "", line 1360, in _find_and_load 2025-10-10T01:34:12.1275074Z (EngineCore_DP0 pid=773) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:12.1275646Z (EngineCore_DP0 pid=773) File "", line 935, in _load_unlocked 2025-10-10T01:34:12.1276208Z (EngineCore_DP0 pid=773) File "", line 999, in exec_module 2025-10-10T01:34:12.1276793Z (EngineCore_DP0 pid=773) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:12.1277518Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:12.1278271Z (EngineCore_DP0 pid=773) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:12.1278993Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:12.1279805Z (EngineCore_DP0 pid=773) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:12.1280582Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:12.1281253Z (EngineCore_DP0 pid=773) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:12.1282038Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:12.1282849Z (EngineCore_DP0 pid=773) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:12.1283322Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1284006Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:12.1284672Z (EngineCore_DP0 pid=773) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:12.1285102Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1285837Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:12.1286534Z (EngineCore_DP0 pid=773) return _is_fa2_supported(device)[0] 2025-10-10T01:34:12.1286948Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1287656Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:12.1288374Z (EngineCore_DP0 pid=773) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:12.1288819Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1289477Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:12.1290104Z (EngineCore_DP0 pid=773) prop = get_device_properties(device) 2025-10-10T01:34:12.1290501Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:12.1291156Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:12.1291896Z (EngineCore_DP0 pid=773) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:12.1292282Z (EngineCore_DP0 pid=773) ^^^^^^^^^^^^ 2025-10-10T01:34:12.1292854Z (EngineCore_DP0 pid=773) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:12.1293404Z (EngineCore_DP0 pid=773) raise RuntimeError( 2025-10-10T01:34:12.1294094Z (EngineCore_DP0 pid=773) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:12.5687259Z FAILED 2025-10-10T01:34:12.5695831Z models/test_transformers.py::test_distributed SKIPPED (Need at least...) 2025-10-10T01:34:12.5713344Z models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ-quantization_kwargs0] INFO 10-10 01:34:12 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'enforce_eager': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model': 'TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ'} 2025-10-10T01:34:12.6408007Z 2025-10-10T01:34:12.6409640Z config.json: 0% 0.00/854 [00:00, 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:20.5022923Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:20.5023627Z 2025-10-10T01:34:20.5024088Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:20.5024609Z 2025-10-10T01:34:20.5025361Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:20.5025966Z 2025-10-10T01:34:20.5026356Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:20.5026733Z 2025-10-10T01:34:20.5027078Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5027413Z 2025-10-10T01:34:20.5028015Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:20.5028687Z 2025-10-10T01:34:20.5029350Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:20.5029819Z 2025-10-10T01:34:20.5030417Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:20.5030931Z 2025-10-10T01:34:20.5031264Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:20.5031596Z 2025-10-10T01:34:20.5031872Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5032153Z 2025-10-10T01:34:20.5032829Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:20.5033723Z 2025-10-10T01:34:20.5034218Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] self._init_executor() 2025-10-10T01:34:20.5034882Z 2025-10-10T01:34:20.5035773Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:20.5036689Z 2025-10-10T01:34:20.5037151Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:20.5037490Z 2025-10-10T01:34:20.5038056Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:20.5038557Z 2025-10-10T01:34:20.5038915Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:20.5039381Z 2025-10-10T01:34:20.5039680Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5039983Z 2025-10-10T01:34:20.5040666Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:20.5041150Z 2025-10-10T01:34:20.5041422Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:20.5041696Z 2025-10-10T01:34:20.5041941Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5042204Z 2025-10-10T01:34:20.5042734Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:20.5043226Z 2025-10-10T01:34:20.5043519Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:20.5043810Z 2025-10-10T01:34:20.5044082Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5044348Z 2025-10-10T01:34:20.5044875Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:20.5045358Z 2025-10-10T01:34:20.5045655Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:20.5045956Z 2025-10-10T01:34:20.5046225Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5046501Z 2025-10-10T01:34:20.5046941Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:20.5047361Z 2025-10-10T01:34:20.5047701Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:20.5048034Z 2025-10-10T01:34:20.5048313Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5048584Z 2025-10-10T01:34:20.5048942Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:20.5049270Z 2025-10-10T01:34:20.5049625Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:20.5049960Z 2025-10-10T01:34:20.5050332Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:20.5050771Z 2025-10-10T01:34:20.5051118Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:20.5051455Z 2025-10-10T01:34:20.5051811Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:20.5052161Z 2025-10-10T01:34:20.5052530Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:20.5052891Z 2025-10-10T01:34:20.5053387Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:20.5054063Z 2025-10-10T01:34:20.5054437Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:20.5054787Z 2025-10-10T01:34:20.5055411Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:20.5055900Z 2025-10-10T01:34:20.5056286Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:20.5056643Z 2025-10-10T01:34:20.5057196Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:20.5057700Z 2025-10-10T01:34:20.5057993Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:20.5058290Z 2025-10-10T01:34:20.5058925Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:20.5059501Z 2025-10-10T01:34:20.5059863Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:20.5060232Z 2025-10-10T01:34:20.5060486Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5060749Z 2025-10-10T01:34:20.5061304Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:20.5061824Z 2025-10-10T01:34:20.5062115Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:20.5062411Z 2025-10-10T01:34:20.5062690Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5062959Z 2025-10-10T01:34:20.5063558Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:20.5064106Z 2025-10-10T01:34:20.5064389Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:20.5064670Z 2025-10-10T01:34:20.5064922Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5065191Z 2025-10-10T01:34:20.5065758Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:20.5066369Z 2025-10-10T01:34:20.5066690Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:20.5067004Z 2025-10-10T01:34:20.5067272Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5067540Z 2025-10-10T01:34:20.5068078Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:20.5068563Z 2025-10-10T01:34:20.5068842Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:20.5069125Z 2025-10-10T01:34:20.5069386Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5069659Z 2025-10-10T01:34:20.5070252Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:20.5070735Z 2025-10-10T01:34:20.5071041Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:20.5071362Z 2025-10-10T01:34:20.5071587Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:20.5071829Z 2025-10-10T01:34:20.5072306Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:20.5072757Z 2025-10-10T01:34:20.5072997Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] raise RuntimeError( 2025-10-10T01:34:20.5073253Z 2025-10-10T01:34:20.5073823Z (EngineCore_DP0 pid=832) ERROR 10-10 01:34:20 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:20.5074511Z (EngineCore_DP0 pid=832) Process EngineCore_DP0: 2025-10-10T01:34:20.5074900Z (EngineCore_DP0 pid=832) Traceback (most recent call last): 2025-10-10T01:34:20.5075537Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:20.5076044Z (EngineCore_DP0 pid=832) self.run() 2025-10-10T01:34:20.5076564Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:20.5077115Z (EngineCore_DP0 pid=832) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:20.5077768Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:20.5078322Z (EngineCore_DP0 pid=832) raise e 2025-10-10T01:34:20.5078907Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:20.5079620Z (EngineCore_DP0 pid=832) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:20.5080051Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5080664Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:20.5081499Z (EngineCore_DP0 pid=832) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:20.5082188Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:20.5082935Z (EngineCore_DP0 pid=832) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:20.5083389Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5084033Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:20.5084801Z (EngineCore_DP0 pid=832) self._init_executor() 2025-10-10T01:34:20.5085467Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:20.5086152Z (EngineCore_DP0 pid=832) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:20.5086874Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:20.5087704Z (EngineCore_DP0 pid=832) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:20.5088210Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5088846Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:20.5089434Z (EngineCore_DP0 pid=832) return func(*args, **kwargs) 2025-10-10T01:34:20.5089805Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5090433Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:20.5091062Z (EngineCore_DP0 pid=832) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:20.5091498Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5092165Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:20.5092807Z (EngineCore_DP0 pid=832) module = importlib.import_module(module_name) 2025-10-10T01:34:20.5093243Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5093825Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:20.5094438Z (EngineCore_DP0 pid=832) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:20.5094919Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5095399Z (EngineCore_DP0 pid=832) File "", line 1387, in _gcd_import 2025-10-10T01:34:20.5095957Z (EngineCore_DP0 pid=832) File "", line 1360, in _find_and_load 2025-10-10T01:34:20.5096781Z (EngineCore_DP0 pid=832) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:20.5097374Z (EngineCore_DP0 pid=832) File "", line 935, in _load_unlocked 2025-10-10T01:34:20.5097943Z (EngineCore_DP0 pid=832) File "", line 999, in exec_module 2025-10-10T01:34:20.5098534Z (EngineCore_DP0 pid=832) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:20.5099272Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:20.5099941Z (EngineCore_DP0 pid=832) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:20.5100788Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:20.5101504Z (EngineCore_DP0 pid=832) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:20.5102274Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:20.5102927Z (EngineCore_DP0 pid=832) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:20.5103748Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:20.5104564Z (EngineCore_DP0 pid=832) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:20.5105063Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5105853Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:20.5106535Z (EngineCore_DP0 pid=832) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:20.5106965Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5107704Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:20.5108398Z (EngineCore_DP0 pid=832) return _is_fa2_supported(device)[0] 2025-10-10T01:34:20.5108802Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5109504Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:20.5110225Z (EngineCore_DP0 pid=832) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:20.5110672Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5111334Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:20.5111961Z (EngineCore_DP0 pid=832) prop = get_device_properties(device) 2025-10-10T01:34:20.5112369Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:20.5113029Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:20.5113683Z (EngineCore_DP0 pid=832) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:20.5114075Z (EngineCore_DP0 pid=832) ^^^^^^^^^^^^ 2025-10-10T01:34:20.5114656Z (EngineCore_DP0 pid=832) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:20.5115277Z (EngineCore_DP0 pid=832) raise RuntimeError( 2025-10-10T01:34:20.5116068Z (EngineCore_DP0 pid=832) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:20.9913383Z FAILED 2025-10-10T01:34:20.9950058Z models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ-quantization_kwargs1] INFO 10-10 01:34:20 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'enforce_eager': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model': 'TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ'} 2025-10-10T01:34:21.0833978Z 2025-10-10T01:34:21.0835754Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:34:21.0836055Z config.json: 1.06kB [00:00, 7.13MB/s] 2025-10-10T01:34:21.1881182Z INFO 10-10 01:34:21 [model.py:551] Resolved architecture: LlamaForCausalLM 2025-10-10T01:34:21.1882278Z INFO 10-10 01:34:21 [model.py:1545] Using max model len 1024 2025-10-10T01:34:21.1886705Z INFO 10-10 01:34:21 [gptq_marlin.py:191] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. 2025-10-10T01:34:21.1887840Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:21.2341755Z INFO 10-10 01:34:21 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:34:21.3179702Z INFO 10-10 01:34:21 [vllm.py:345] Cudagraph is disabled under eager mode 2025-10-10T01:34:21.3573729Z 2025-10-10T01:34:21.3575431Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:34:21.3575807Z tokenizer_config.json: 1.42kB [00:00, 11.1MB/s] 2025-10-10T01:34:21.4954199Z 2025-10-10T01:34:21.6509825Z tokenizer.model: 0% 0.00/500k [00:00, 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:22.1185141Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:22.1185569Z 2025-10-10T01:34:22.1185973Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:22.1186339Z 2025-10-10T01:34:22.1187010Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:22.1187634Z 2025-10-10T01:34:22.1188013Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:22.1188696Z 2025-10-10T01:34:22.1189066Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1189413Z 2025-10-10T01:34:22.1190008Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:22.1190540Z 2025-10-10T01:34:22.1190870Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:22.1191180Z 2025-10-10T01:34:22.1191822Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:22.1192271Z 2025-10-10T01:34:22.1192613Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:22.1192920Z 2025-10-10T01:34:22.1193198Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1193476Z 2025-10-10T01:34:22.1194023Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:22.1194486Z 2025-10-10T01:34:22.1194737Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] self._init_executor() 2025-10-10T01:34:22.1194991Z 2025-10-10T01:34:22.1195524Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:22.1196450Z 2025-10-10T01:34:22.1196827Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:22.1197154Z 2025-10-10T01:34:22.1197973Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:22.1198719Z 2025-10-10T01:34:22.1199347Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:22.1199764Z 2025-10-10T01:34:22.1200184Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1200570Z 2025-10-10T01:34:22.1201329Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:22.1202020Z 2025-10-10T01:34:22.1202329Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:22.1202604Z 2025-10-10T01:34:22.1202863Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1203120Z 2025-10-10T01:34:22.1203630Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:22.1204104Z 2025-10-10T01:34:22.1204389Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:22.1204679Z 2025-10-10T01:34:22.1204942Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1205213Z 2025-10-10T01:34:22.1205852Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:22.1206351Z 2025-10-10T01:34:22.1206664Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:22.1206967Z 2025-10-10T01:34:22.1207244Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1207518Z 2025-10-10T01:34:22.1207965Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:22.1208372Z 2025-10-10T01:34:22.1208710Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:22.1209037Z 2025-10-10T01:34:22.1209320Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1209595Z 2025-10-10T01:34:22.1209932Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:22.1210277Z 2025-10-10T01:34:22.1210618Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:22.1210951Z 2025-10-10T01:34:22.1211319Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:22.1211674Z 2025-10-10T01:34:22.1212012Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:22.1212338Z 2025-10-10T01:34:22.1212703Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:22.1213046Z 2025-10-10T01:34:22.1213429Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:22.1213786Z 2025-10-10T01:34:22.1214280Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:22.1214744Z 2025-10-10T01:34:22.1215088Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:22.1215422Z 2025-10-10T01:34:22.1215932Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:22.1216523Z 2025-10-10T01:34:22.1216892Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:22.1217247Z 2025-10-10T01:34:22.1217802Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:22.1218327Z 2025-10-10T01:34:22.1218626Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:22.1218932Z 2025-10-10T01:34:22.1219591Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:22.1220176Z 2025-10-10T01:34:22.1220549Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:22.1220973Z 2025-10-10T01:34:22.1221242Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1221513Z 2025-10-10T01:34:22.1222084Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:22.1222610Z 2025-10-10T01:34:22.1222902Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:22.1223198Z 2025-10-10T01:34:22.1223466Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1223741Z 2025-10-10T01:34:22.1224338Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:22.1224891Z 2025-10-10T01:34:22.1225176Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:22.1225462Z 2025-10-10T01:34:22.1225727Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1225989Z 2025-10-10T01:34:22.1226572Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:22.1227097Z 2025-10-10T01:34:22.1227418Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:22.1227724Z 2025-10-10T01:34:22.1227994Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1228271Z 2025-10-10T01:34:22.1228788Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:22.1229272Z 2025-10-10T01:34:22.1229548Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:22.1229868Z 2025-10-10T01:34:22.1230168Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1230494Z 2025-10-10T01:34:22.1231118Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:22.1231677Z 2025-10-10T01:34:22.1232178Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:22.1232487Z 2025-10-10T01:34:22.1232718Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:22.1232958Z 2025-10-10T01:34:22.1233445Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:22.1233890Z 2025-10-10T01:34:22.1234130Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] raise RuntimeError( 2025-10-10T01:34:22.1234394Z 2025-10-10T01:34:22.1234964Z (EngineCore_DP0 pid=839) ERROR 10-10 01:34:22 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:22.1235657Z (EngineCore_DP0 pid=839) Process EngineCore_DP0: 2025-10-10T01:34:22.1236056Z (EngineCore_DP0 pid=839) Traceback (most recent call last): 2025-10-10T01:34:22.1236756Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:22.1237289Z (EngineCore_DP0 pid=839) self.run() 2025-10-10T01:34:22.1237811Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:22.1238362Z (EngineCore_DP0 pid=839) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:22.1239122Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:22.1239697Z (EngineCore_DP0 pid=839) raise e 2025-10-10T01:34:22.1240285Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:22.1240927Z (EngineCore_DP0 pid=839) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:22.1241358Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1241974Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:22.1242603Z (EngineCore_DP0 pid=839) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:22.1243250Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:22.1243857Z (EngineCore_DP0 pid=839) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:22.1244296Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1244958Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:22.1245559Z (EngineCore_DP0 pid=839) self._init_executor() 2025-10-10T01:34:22.1246219Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:22.1246901Z (EngineCore_DP0 pid=839) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:22.1247619Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:22.1248320Z (EngineCore_DP0 pid=839) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:22.1248812Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1249534Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:22.1250113Z (EngineCore_DP0 pid=839) return func(*args, **kwargs) 2025-10-10T01:34:22.1250493Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1251116Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:22.1251735Z (EngineCore_DP0 pid=839) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:22.1252152Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1252809Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:22.1253471Z (EngineCore_DP0 pid=839) module = importlib.import_module(module_name) 2025-10-10T01:34:22.1253992Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1254578Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:22.1255194Z (EngineCore_DP0 pid=839) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:22.1255675Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1256162Z (EngineCore_DP0 pid=839) File "", line 1387, in _gcd_import 2025-10-10T01:34:22.1256707Z (EngineCore_DP0 pid=839) File "", line 1360, in _find_and_load 2025-10-10T01:34:22.1257284Z (EngineCore_DP0 pid=839) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:22.1257873Z (EngineCore_DP0 pid=839) File "", line 935, in _load_unlocked 2025-10-10T01:34:22.1258441Z (EngineCore_DP0 pid=839) File "", line 999, in exec_module 2025-10-10T01:34:22.1259038Z (EngineCore_DP0 pid=839) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:22.1259755Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:22.1260438Z (EngineCore_DP0 pid=839) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:22.1261155Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:22.1261873Z (EngineCore_DP0 pid=839) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:22.1262659Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:22.1263323Z (EngineCore_DP0 pid=839) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:22.1264096Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:22.1264905Z (EngineCore_DP0 pid=839) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:22.1265376Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1266062Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:22.1266840Z (EngineCore_DP0 pid=839) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:22.1267274Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1268009Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:22.1268703Z (EngineCore_DP0 pid=839) return _is_fa2_supported(device)[0] 2025-10-10T01:34:22.1269107Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1269854Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:22.1270690Z (EngineCore_DP0 pid=839) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:22.1271218Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1272027Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:22.1272659Z (EngineCore_DP0 pid=839) prop = get_device_properties(device) 2025-10-10T01:34:22.1273061Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:22.1273702Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:22.1274369Z (EngineCore_DP0 pid=839) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:22.1274752Z (EngineCore_DP0 pid=839) ^^^^^^^^^^^^ 2025-10-10T01:34:22.1275326Z (EngineCore_DP0 pid=839) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:22.1275887Z (EngineCore_DP0 pid=839) raise RuntimeError( 2025-10-10T01:34:22.1276566Z (EngineCore_DP0 pid=839) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:22.5536018Z FAILED 2025-10-10T01:34:22.5554598Z models/test_transformers.py::test_quantization[5-32-meta-llama/Llama-3.2-1B-Instruct-quantization_kwargs2] INFO 10-10 01:34:22 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'quantization': 'bitsandbytes', 'enforce_eager': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model': 'meta-llama/Llama-3.2-1B-Instruct'} 2025-10-10T01:34:22.6346586Z INFO 10-10 01:34:22 [model.py:551] Resolved architecture: LlamaForCausalLM 2025-10-10T01:34:22.6347638Z INFO 10-10 01:34:22 [model.py:1545] Using max model len 1024 2025-10-10T01:34:22.6348970Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:22.6852651Z INFO 10-10 01:34:22 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:34:22.6856255Z INFO 10-10 01:34:22 [vllm.py:345] Cudagraph is disabled under eager mode 2025-10-10T01:34:23.1598890Z (EngineCore_DP0 pid=845) INFO 10-10 01:34:23 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:34:23.1631149Z (EngineCore_DP0 pid=845) INFO 10-10 01:34:23 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='meta-llama/Llama-3.2-1B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.2-1B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1024, download_dir=None, load_format=bitsandbytes, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=meta-llama/Llama-3.2-1B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 0, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:23.3007696Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:23.3008096Z 2025-10-10T01:34:23.3008474Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:23.3008834Z 2025-10-10T01:34:23.3009516Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:23.3010136Z 2025-10-10T01:34:23.3010514Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:23.3010890Z 2025-10-10T01:34:23.3011229Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3011580Z 2025-10-10T01:34:23.3012072Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:23.3012556Z 2025-10-10T01:34:23.3013011Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:23.3013351Z 2025-10-10T01:34:23.3013853Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:23.3014298Z 2025-10-10T01:34:23.3014611Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:23.3014913Z 2025-10-10T01:34:23.3015186Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3015459Z 2025-10-10T01:34:23.3016068Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:23.3016801Z 2025-10-10T01:34:23.3017197Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] self._init_executor() 2025-10-10T01:34:23.3017577Z 2025-10-10T01:34:23.3018706Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:23.3019243Z 2025-10-10T01:34:23.3019592Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:23.3019912Z 2025-10-10T01:34:23.3020454Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:23.3020971Z 2025-10-10T01:34:23.3021320Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:23.3021652Z 2025-10-10T01:34:23.3021933Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3022221Z 2025-10-10T01:34:23.3023502Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:23.3023975Z 2025-10-10T01:34:23.3024248Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:23.3024522Z 2025-10-10T01:34:23.3024781Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3025037Z 2025-10-10T01:34:23.3025565Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:23.3026037Z 2025-10-10T01:34:23.3026322Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:23.3026615Z 2025-10-10T01:34:23.3026876Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3027171Z 2025-10-10T01:34:23.3027701Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:23.3028185Z 2025-10-10T01:34:23.3028487Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:23.3028780Z 2025-10-10T01:34:23.3029059Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3029328Z 2025-10-10T01:34:23.3029762Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:23.3030178Z 2025-10-10T01:34:23.3030517Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:23.3030840Z 2025-10-10T01:34:23.3031118Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3031393Z 2025-10-10T01:34:23.3031745Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:23.3032076Z 2025-10-10T01:34:23.3032417Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:23.3032747Z 2025-10-10T01:34:23.3033119Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:23.3033560Z 2025-10-10T01:34:23.3033907Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:23.3034233Z 2025-10-10T01:34:23.3034597Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:23.3034940Z 2025-10-10T01:34:23.3035306Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:23.3035665Z 2025-10-10T01:34:23.3036154Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:23.3036615Z 2025-10-10T01:34:23.3036962Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:23.3037310Z 2025-10-10T01:34:23.3037913Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:23.3038404Z 2025-10-10T01:34:23.3038772Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:23.3039248Z 2025-10-10T01:34:23.3039811Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:23.3040318Z 2025-10-10T01:34:23.3040625Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:23.3040936Z 2025-10-10T01:34:23.3041585Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:23.3042167Z 2025-10-10T01:34:23.3042536Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:23.3042889Z 2025-10-10T01:34:23.3043140Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3043408Z 2025-10-10T01:34:23.3043980Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:23.3044507Z 2025-10-10T01:34:23.3044798Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:23.3045096Z 2025-10-10T01:34:23.3045363Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3045634Z 2025-10-10T01:34:23.3046242Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:23.3046792Z 2025-10-10T01:34:23.3047074Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:23.3047355Z 2025-10-10T01:34:23.3047614Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3047874Z 2025-10-10T01:34:23.3048451Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:23.3049058Z 2025-10-10T01:34:23.3049382Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:23.3049694Z 2025-10-10T01:34:23.3049961Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3050248Z 2025-10-10T01:34:23.3050772Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:23.3051258Z 2025-10-10T01:34:23.3051531Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:23.3051814Z 2025-10-10T01:34:23.3052076Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3052340Z 2025-10-10T01:34:23.3052930Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:23.3053417Z 2025-10-10T01:34:23.3053732Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:23.3054031Z 2025-10-10T01:34:23.3054254Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:23.3054491Z 2025-10-10T01:34:23.3054967Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:23.3055433Z 2025-10-10T01:34:23.3055673Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] raise RuntimeError( 2025-10-10T01:34:23.3055932Z 2025-10-10T01:34:23.3056492Z (EngineCore_DP0 pid=845) ERROR 10-10 01:34:23 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:23.3057183Z (EngineCore_DP0 pid=845) Process EngineCore_DP0: 2025-10-10T01:34:23.3057571Z (EngineCore_DP0 pid=845) Traceback (most recent call last): 2025-10-10T01:34:23.3058171Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:23.3058677Z (EngineCore_DP0 pid=845) self.run() 2025-10-10T01:34:23.3059203Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:23.3059755Z (EngineCore_DP0 pid=845) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:23.3060433Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:23.3060979Z (EngineCore_DP0 pid=845) raise e 2025-10-10T01:34:23.3061555Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:23.3062192Z (EngineCore_DP0 pid=845) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:23.3062630Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3063239Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:23.3063874Z (EngineCore_DP0 pid=845) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:23.3064521Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:23.3065225Z (EngineCore_DP0 pid=845) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:23.3065678Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3066322Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:23.3066906Z (EngineCore_DP0 pid=845) self._init_executor() 2025-10-10T01:34:23.3067554Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:23.3068231Z (EngineCore_DP0 pid=845) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:23.3068946Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:23.3069654Z (EngineCore_DP0 pid=845) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:23.3070210Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3070854Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:23.3071446Z (EngineCore_DP0 pid=845) return func(*args, **kwargs) 2025-10-10T01:34:23.3071837Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3072467Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:23.3073087Z (EngineCore_DP0 pid=845) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:23.3073498Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3074168Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:23.3074814Z (EngineCore_DP0 pid=845) module = importlib.import_module(module_name) 2025-10-10T01:34:23.3075247Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3075824Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:23.3076434Z (EngineCore_DP0 pid=845) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:23.3076905Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3077388Z (EngineCore_DP0 pid=845) File "", line 1387, in _gcd_import 2025-10-10T01:34:23.3077929Z (EngineCore_DP0 pid=845) File "", line 1360, in _find_and_load 2025-10-10T01:34:23.3078516Z (EngineCore_DP0 pid=845) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:23.3079151Z (EngineCore_DP0 pid=845) File "", line 935, in _load_unlocked 2025-10-10T01:34:23.3079714Z (EngineCore_DP0 pid=845) File "", line 999, in exec_module 2025-10-10T01:34:23.3080312Z (EngineCore_DP0 pid=845) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:23.3081040Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:23.3081719Z (EngineCore_DP0 pid=845) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:23.3082429Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:23.3083234Z (EngineCore_DP0 pid=845) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:23.3084006Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:23.3084669Z (EngineCore_DP0 pid=845) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:23.3085449Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:23.3086257Z (EngineCore_DP0 pid=845) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:23.3086724Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3087498Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:23.3088185Z (EngineCore_DP0 pid=845) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:23.3088616Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3089351Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:23.3090039Z (EngineCore_DP0 pid=845) return _is_fa2_supported(device)[0] 2025-10-10T01:34:23.3090433Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3091138Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:23.3091855Z (EngineCore_DP0 pid=845) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:23.3092297Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3092960Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:23.3093578Z (EngineCore_DP0 pid=845) prop = get_device_properties(device) 2025-10-10T01:34:23.3093986Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:23.3094631Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:23.3095278Z (EngineCore_DP0 pid=845) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:23.3095664Z (EngineCore_DP0 pid=845) ^^^^^^^^^^^^ 2025-10-10T01:34:23.3096416Z (EngineCore_DP0 pid=845) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:23.3096993Z (EngineCore_DP0 pid=845) raise RuntimeError( 2025-10-10T01:34:23.3097675Z (EngineCore_DP0 pid=845) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:23.7296759Z FAILED 2025-10-10T01:34:23.7311645Z models/test_transformers.py::test_embed_loading[Qwen/Qwen3-Embedding-0.6B] INFO 10-10 01:34:23 [utils.py:233] non-default args: {'runner': 'pooling', 'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'enforce_eager': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model_impl': 'transformers', 'model': 'Qwen/Qwen3-Embedding-0.6B'} 2025-10-10T01:34:23.8116833Z 2025-10-10T01:34:23.8118437Z config.json: 0% 0.00/727 [00:00, 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:31.8067454Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:31.8068019Z 2025-10-10T01:34:31.8068417Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:31.8068952Z 2025-10-10T01:34:31.8069886Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:31.8070385Z 2025-10-10T01:34:31.8070707Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:31.8071017Z 2025-10-10T01:34:31.8071312Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8071602Z 2025-10-10T01:34:31.8072107Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:31.8072556Z 2025-10-10T01:34:31.8072983Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:31.8073373Z 2025-10-10T01:34:31.8073878Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:31.8074324Z 2025-10-10T01:34:31.8074632Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:31.8074939Z 2025-10-10T01:34:31.8075237Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8075517Z 2025-10-10T01:34:31.8076021Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:31.8076490Z 2025-10-10T01:34:31.8076756Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] self._init_executor() 2025-10-10T01:34:31.8077228Z 2025-10-10T01:34:31.8078303Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:31.8079298Z 2025-10-10T01:34:31.8079944Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:31.8080537Z 2025-10-10T01:34:31.8081545Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:31.8082091Z 2025-10-10T01:34:31.8082466Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:31.8082801Z 2025-10-10T01:34:31.8083092Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8083562Z 2025-10-10T01:34:31.8084104Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:31.8084558Z 2025-10-10T01:34:31.8084825Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:31.8085098Z 2025-10-10T01:34:31.8085347Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8085610Z 2025-10-10T01:34:31.8086118Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:31.8086598Z 2025-10-10T01:34:31.8086879Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:31.8087170Z 2025-10-10T01:34:31.8087531Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8087803Z 2025-10-10T01:34:31.8088333Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:31.8088811Z 2025-10-10T01:34:31.8089122Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:31.8089419Z 2025-10-10T01:34:31.8089692Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8089976Z 2025-10-10T01:34:31.8090409Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:31.8090827Z 2025-10-10T01:34:31.8091167Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:31.8091500Z 2025-10-10T01:34:31.8091780Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8092055Z 2025-10-10T01:34:31.8092393Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:31.8092716Z 2025-10-10T01:34:31.8093063Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:31.8093397Z 2025-10-10T01:34:31.8093780Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:31.8094143Z 2025-10-10T01:34:31.8094503Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:31.8094845Z 2025-10-10T01:34:31.8095215Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:31.8095567Z 2025-10-10T01:34:31.8106412Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:31.8106860Z 2025-10-10T01:34:31.8107445Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:31.8107963Z 2025-10-10T01:34:31.8108330Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:31.8108880Z 2025-10-10T01:34:31.8109428Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:31.8109921Z 2025-10-10T01:34:31.8110294Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:31.8110674Z 2025-10-10T01:34:31.8111239Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:31.8111744Z 2025-10-10T01:34:31.8112055Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:31.8112353Z 2025-10-10T01:34:31.8113112Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:31.8113705Z 2025-10-10T01:34:31.8114085Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:31.8114439Z 2025-10-10T01:34:31.8114701Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8114963Z 2025-10-10T01:34:31.8115525Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:31.8116046Z 2025-10-10T01:34:31.8116340Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:31.8116640Z 2025-10-10T01:34:31.8116929Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8117230Z 2025-10-10T01:34:31.8117860Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:31.8118418Z 2025-10-10T01:34:31.8118696Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:31.8118978Z 2025-10-10T01:34:31.8119371Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8119634Z 2025-10-10T01:34:31.8120219Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:31.8120759Z 2025-10-10T01:34:31.8121089Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:31.8121398Z 2025-10-10T01:34:31.8121667Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8121947Z 2025-10-10T01:34:31.8122468Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:31.8122956Z 2025-10-10T01:34:31.8123232Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:31.8123523Z 2025-10-10T01:34:31.8123777Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8124039Z 2025-10-10T01:34:31.8124701Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:31.8125189Z 2025-10-10T01:34:31.8125506Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:31.8125808Z 2025-10-10T01:34:31.8126034Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:31.8126271Z 2025-10-10T01:34:31.8126747Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:31.8127200Z 2025-10-10T01:34:31.8127443Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] raise RuntimeError( 2025-10-10T01:34:31.8127711Z 2025-10-10T01:34:31.8128359Z (EngineCore_DP0 pid=903) ERROR 10-10 01:34:31 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:31.8129073Z (EngineCore_DP0 pid=903) Process EngineCore_DP0: 2025-10-10T01:34:31.8129470Z (EngineCore_DP0 pid=903) Traceback (most recent call last): 2025-10-10T01:34:31.8130109Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:31.8130622Z (EngineCore_DP0 pid=903) self.run() 2025-10-10T01:34:31.8131141Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:31.8131712Z (EngineCore_DP0 pid=903) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:31.8132378Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:31.8132932Z (EngineCore_DP0 pid=903) raise e 2025-10-10T01:34:31.8133525Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:31.8134159Z (EngineCore_DP0 pid=903) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:31.8134592Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8135219Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:31.8135863Z (EngineCore_DP0 pid=903) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:31.8136525Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:31.8137150Z (EngineCore_DP0 pid=903) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:31.8137600Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8138251Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:31.8138834Z (EngineCore_DP0 pid=903) self._init_executor() 2025-10-10T01:34:31.8139500Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:31.8140189Z (EngineCore_DP0 pid=903) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:31.8140910Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:31.8141714Z (EngineCore_DP0 pid=903) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:31.8142224Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8142878Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:31.8143469Z (EngineCore_DP0 pid=903) return func(*args, **kwargs) 2025-10-10T01:34:31.8143839Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8144477Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:31.8145126Z (EngineCore_DP0 pid=903) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:31.8145551Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8146305Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:31.8146961Z (EngineCore_DP0 pid=903) module = importlib.import_module(module_name) 2025-10-10T01:34:31.8147410Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8147997Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:31.8148622Z (EngineCore_DP0 pid=903) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:31.8149120Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8149603Z (EngineCore_DP0 pid=903) File "", line 1387, in _gcd_import 2025-10-10T01:34:31.8150160Z (EngineCore_DP0 pid=903) File "", line 1360, in _find_and_load 2025-10-10T01:34:31.8150745Z (EngineCore_DP0 pid=903) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:31.8151319Z (EngineCore_DP0 pid=903) File "", line 935, in _load_unlocked 2025-10-10T01:34:31.8151884Z (EngineCore_DP0 pid=903) File "", line 999, in exec_module 2025-10-10T01:34:31.8152483Z (EngineCore_DP0 pid=903) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:31.8153203Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:31.8153876Z (EngineCore_DP0 pid=903) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:31.8154589Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:31.8155317Z (EngineCore_DP0 pid=903) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:31.8156094Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:31.8156752Z (EngineCore_DP0 pid=903) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:31.8157536Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:31.8158347Z (EngineCore_DP0 pid=903) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:31.8158824Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8159722Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:31.8160420Z (EngineCore_DP0 pid=903) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:31.8160852Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8161605Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:31.8162301Z (EngineCore_DP0 pid=903) return _is_fa2_supported(device)[0] 2025-10-10T01:34:31.8162721Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8163433Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:31.8164243Z (EngineCore_DP0 pid=903) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:31.8164719Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8165392Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:31.8166024Z (EngineCore_DP0 pid=903) prop = get_device_properties(device) 2025-10-10T01:34:31.8166439Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:31.8167084Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:31.8167741Z (EngineCore_DP0 pid=903) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:31.8168136Z (EngineCore_DP0 pid=903) ^^^^^^^^^^^^ 2025-10-10T01:34:31.8168719Z (EngineCore_DP0 pid=903) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:31.8169293Z (EngineCore_DP0 pid=903) raise RuntimeError( 2025-10-10T01:34:31.8169983Z (EngineCore_DP0 pid=903) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:32.2623890Z FAILED 2025-10-10T01:34:32.2643851Z models/test_transformers.py::test_embed_loading[meta-llama/Llama-3.2-1B-Instruct] INFO 10-10 01:34:32 [utils.py:233] non-default args: {'runner': 'pooling', 'trust_remote_code': True, 'seed': 0, 'max_model_len': 1024, 'block_size': 16, 'disable_log_stats': True, 'enforce_eager': True, 'enable_chunked_prefill': False, 'compilation_config': {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}, 'model_impl': 'transformers', 'model': 'meta-llama/Llama-3.2-1B-Instruct'} 2025-10-10T01:34:32.3355484Z INFO 10-10 01:34:32 [model.py:551] Resolved architecture: TransformersEmbeddingModel 2025-10-10T01:34:32.3356299Z INFO 10-10 01:34:32 [model.py:1545] Using max model len 1024 2025-10-10T01:34:32.3357764Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:32.3587348Z INFO 10-10 01:34:32 [arg_utils.py:1583] (Disabling) prefix caching by default 2025-10-10T01:34:32.3592926Z INFO 10-10 01:34:32 [vllm.py:345] Cudagraph is disabled under eager mode 2025-10-10T01:34:32.3593710Z INFO 10-10 01:34:32 [vllm.py:404] Only "last" pooling supports chunked prefill and prefix caching; disabling both. 2025-10-10T01:34:32.8298483Z (EngineCore_DP0 pid=909) INFO 10-10 01:34:32 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:34:32.8333478Z (EngineCore_DP0 pid=909) INFO 10-10 01:34:32 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='meta-llama/Llama-3.2-1B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.2-1B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1024, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=meta-llama/Llama-3.2-1B-Instruct, enable_prefix_caching=False, chunked_prefill_enabled=False, pooler_config=PoolerConfig(pooling_type='CLS', normalize=None, dimensions=None, enable_chunked_processing=None, max_embed_len=None, activation=None, logit_bias=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={'level': 0, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 4, 'local_cache_dir': None} 2025-10-10T01:34:32.9720631Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] EngineCore failed to start. 2025-10-10T01:34:32.9721080Z 2025-10-10T01:34:32.9721647Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] Traceback (most recent call last): 2025-10-10T01:34:32.9722026Z 2025-10-10T01:34:32.9722719Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:32.9723343Z 2025-10-10T01:34:32.9723727Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:32.9724094Z 2025-10-10T01:34:32.9724438Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9724799Z 2025-10-10T01:34:32.9725417Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:32.9725965Z 2025-10-10T01:34:32.9726374Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:32.9726763Z 2025-10-10T01:34:32.9727529Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:32.9728097Z 2025-10-10T01:34:32.9728495Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:32.9728870Z 2025-10-10T01:34:32.9729208Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9729890Z 2025-10-10T01:34:32.9730422Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:32.9730898Z 2025-10-10T01:34:32.9731147Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] self._init_executor() 2025-10-10T01:34:32.9731620Z 2025-10-10T01:34:32.9732482Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:32.9733343Z 2025-10-10T01:34:32.9733772Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:32.9734187Z 2025-10-10T01:34:32.9735418Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:32.9736215Z 2025-10-10T01:34:32.9736614Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:32.9736964Z 2025-10-10T01:34:32.9737254Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9737541Z 2025-10-10T01:34:32.9738085Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:32.9738542Z 2025-10-10T01:34:32.9738813Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:34:32.9739080Z 2025-10-10T01:34:32.9739332Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9739595Z 2025-10-10T01:34:32.9740117Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:32.9740596Z 2025-10-10T01:34:32.9740875Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:32.9741164Z 2025-10-10T01:34:32.9741434Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9741707Z 2025-10-10T01:34:32.9742227Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:32.9742706Z 2025-10-10T01:34:32.9743015Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:34:32.9743312Z 2025-10-10T01:34:32.9743590Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9743863Z 2025-10-10T01:34:32.9744304Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:32.9744717Z 2025-10-10T01:34:32.9745062Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:32.9745405Z 2025-10-10T01:34:32.9745686Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9745965Z 2025-10-10T01:34:32.9746303Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:34:32.9746796Z 2025-10-10T01:34:32.9747161Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:34:32.9747533Z 2025-10-10T01:34:32.9747917Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:32.9748278Z 2025-10-10T01:34:32.9748624Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:34:32.9748959Z 2025-10-10T01:34:32.9749323Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:34:32.9749671Z 2025-10-10T01:34:32.9750059Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:32.9750422Z 2025-10-10T01:34:32.9751068Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:32.9751549Z 2025-10-10T01:34:32.9751902Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:32.9752246Z 2025-10-10T01:34:32.9752767Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:32.9753254Z 2025-10-10T01:34:32.9753619Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:32.9753986Z 2025-10-10T01:34:32.9754532Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:32.9755050Z 2025-10-10T01:34:32.9755341Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:34:32.9755634Z 2025-10-10T01:34:32.9756271Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:32.9756842Z 2025-10-10T01:34:32.9757209Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:32.9757559Z 2025-10-10T01:34:32.9757815Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9758079Z 2025-10-10T01:34:32.9758637Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:32.9759295Z 2025-10-10T01:34:32.9759591Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:34:32.9759904Z 2025-10-10T01:34:32.9760173Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9760448Z 2025-10-10T01:34:32.9761043Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:32.9761593Z 2025-10-10T01:34:32.9761869Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:34:32.9762243Z 2025-10-10T01:34:32.9762515Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9762786Z 2025-10-10T01:34:32.9763365Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:32.9763889Z 2025-10-10T01:34:32.9764209Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:32.9764537Z 2025-10-10T01:34:32.9764809Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9765086Z 2025-10-10T01:34:32.9765619Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:32.9766126Z 2025-10-10T01:34:32.9766483Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:34:32.9766779Z 2025-10-10T01:34:32.9767040Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9767305Z 2025-10-10T01:34:32.9767826Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:32.9768304Z 2025-10-10T01:34:32.9768616Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:34:32.9768921Z 2025-10-10T01:34:32.9769146Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:34:32.9769387Z 2025-10-10T01:34:32.9769880Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:32.9770334Z 2025-10-10T01:34:32.9770578Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] raise RuntimeError( 2025-10-10T01:34:32.9770849Z 2025-10-10T01:34:32.9771414Z (EngineCore_DP0 pid=909) ERROR 10-10 01:34:32 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:32.9772102Z (EngineCore_DP0 pid=909) Process EngineCore_DP0: 2025-10-10T01:34:32.9772501Z (EngineCore_DP0 pid=909) Traceback (most recent call last): 2025-10-10T01:34:32.9773108Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:34:32.9773617Z (EngineCore_DP0 pid=909) self.run() 2025-10-10T01:34:32.9774143Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:34:32.9774703Z (EngineCore_DP0 pid=909) self._target(*self._args, **self._kwargs) 2025-10-10T01:34:32.9775371Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:34:32.9775915Z (EngineCore_DP0 pid=909) raise e 2025-10-10T01:34:32.9776507Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:34:32.9777146Z (EngineCore_DP0 pid=909) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:34:32.9777589Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9778307Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:34:32.9778979Z (EngineCore_DP0 pid=909) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:34:32.9779682Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:34:32.9780305Z (EngineCore_DP0 pid=909) self.model_executor = executor_class(vllm_config) 2025-10-10T01:34:32.9780765Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9781419Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:34:32.9782013Z (EngineCore_DP0 pid=909) self._init_executor() 2025-10-10T01:34:32.9782757Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:34:32.9783455Z (EngineCore_DP0 pid=909) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:34:32.9784182Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:34:32.9784898Z (EngineCore_DP0 pid=909) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:34:32.9785398Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9786036Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:34:32.9786616Z (EngineCore_DP0 pid=909) return func(*args, **kwargs) 2025-10-10T01:34:32.9786998Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9787645Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:34:32.9788277Z (EngineCore_DP0 pid=909) worker_class = resolve_obj_by_qualname( 2025-10-10T01:34:32.9788695Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9789351Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:34:32.9790006Z (EngineCore_DP0 pid=909) module = importlib.import_module(module_name) 2025-10-10T01:34:32.9790449Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9791027Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:34:32.9791650Z (EngineCore_DP0 pid=909) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:34:32.9792125Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9792631Z (EngineCore_DP0 pid=909) File "", line 1387, in _gcd_import 2025-10-10T01:34:32.9793181Z (EngineCore_DP0 pid=909) File "", line 1360, in _find_and_load 2025-10-10T01:34:32.9793760Z (EngineCore_DP0 pid=909) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:34:32.9794333Z (EngineCore_DP0 pid=909) File "", line 935, in _load_unlocked 2025-10-10T01:34:32.9794897Z (EngineCore_DP0 pid=909) File "", line 999, in exec_module 2025-10-10T01:34:32.9795589Z (EngineCore_DP0 pid=909) File "", line 488, in _call_with_frames_removed 2025-10-10T01:34:32.9796532Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:34:32.9797218Z (EngineCore_DP0 pid=909) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:34:32.9797945Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:34:32.9798665Z (EngineCore_DP0 pid=909) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:34:32.9799482Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:34:32.9800157Z (EngineCore_DP0 pid=909) class FlashAttentionMetadataBuilder( 2025-10-10T01:34:32.9801086Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:34:32.9801928Z (EngineCore_DP0 pid=909) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:34:32.9802406Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9803102Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:34:32.9803787Z (EngineCore_DP0 pid=909) if not is_fa_version_supported(fa_version): 2025-10-10T01:34:32.9804231Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9804969Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:34:32.9805677Z (EngineCore_DP0 pid=909) return _is_fa2_supported(device)[0] 2025-10-10T01:34:32.9806084Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9806793Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:34:32.9807523Z (EngineCore_DP0 pid=909) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:34:32.9807974Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9808639Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:34:32.9809273Z (EngineCore_DP0 pid=909) prop = get_device_properties(device) 2025-10-10T01:34:32.9809689Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:34:32.9810348Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:34:32.9811005Z (EngineCore_DP0 pid=909) _lazy_init() # will define _get_device_properties 2025-10-10T01:34:32.9811397Z (EngineCore_DP0 pid=909) ^^^^^^^^^^^^ 2025-10-10T01:34:32.9811987Z (EngineCore_DP0 pid=909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:34:32.9812548Z (EngineCore_DP0 pid=909) raise RuntimeError( 2025-10-10T01:34:32.9813238Z (EngineCore_DP0 pid=909) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:34:33.4318288Z FAILED 2025-10-10T01:34:33.7434096Z models/test_transformers.py::test_pooling[TransformersEmbeddingModel] SKIPPED 2025-10-10T01:34:34.0256192Z models/test_transformers.py::test_pooling[TransformersForSequenceClassification] SKIPPED 2025-10-10T01:34:34.0256657Z 2025-10-10T01:34:34.0256798Z =================================== FAILURES =================================== 2025-10-10T01:34:34.0257376Z __________ test_models[meta-llama/Llama-3.2-1B-Instruct-transformers] __________ 2025-10-10T01:34:34.0257732Z 2025-10-10T01:34:34.0257858Z hf_runner = 2025-10-10T01:34:34.0258204Z vllm_runner = 2025-10-10T01:34:34.0259225Z example_prompts = ['vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n', 'Briefly describe the majo...me.\n', 'Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.\n', ...] 2025-10-10T01:34:34.0260368Z model = 'meta-llama/Llama-3.2-1B-Instruct', model_impl = 'transformers' 2025-10-10T01:34:34.0260964Z 2025-10-10T01:34:34.0613924Z @pytest.mark.skipif( 2025-10-10T01:34:34.0614385Z current_platform.is_rocm(), 2025-10-10T01:34:34.0615095Z reason="Llama-3.2-1B-Instruct, Ilama-3.2-1B produce memory access fault.") 2025-10-10T01:34:34.0615786Z @pytest.mark.parametrize( 2025-10-10T01:34:34.0616293Z "model,model_impl", 2025-10-10T01:34:34.0616701Z [ 2025-10-10T01:34:34.0617354Z ("meta-llama/Llama-3.2-1B-Instruct", "transformers"), 2025-10-10T01:34:34.0618325Z ("hmellor/Ilama-3.2-1B", "auto"), # CUSTOM CODE 2025-10-10T01:34:34.0619300Z ("allenai/OLMoE-1B-7B-0924", "transformers"), # MoE 2025-10-10T01:34:34.0620005Z ]) # trust_remote_code=True by default 2025-10-10T01:34:34.0620470Z def test_models( 2025-10-10T01:34:34.0620907Z hf_runner: type[HfRunner], 2025-10-10T01:34:34.0621376Z vllm_runner: type[VllmRunner], 2025-10-10T01:34:34.0621803Z example_prompts: list[str], 2025-10-10T01:34:34.0622154Z model: str, 2025-10-10T01:34:34.0622455Z model_impl: str, 2025-10-10T01:34:34.0622738Z ) -> None: 2025-10-10T01:34:34.0623078Z import transformers 2025-10-10T01:34:34.0623649Z from packaging.version import Version 2025-10-10T01:34:34.0624163Z installed = Version(transformers.__version__) 2025-10-10T01:34:34.0624610Z required = Version("4.57.0.dev0") 2025-10-10T01:34:34.0625240Z if model == "allenai/OLMoE-1B-7B-0924" and installed < required: 2025-10-10T01:34:34.0625946Z pytest.skip("MoE models with the Transformers backend require " 2025-10-10T01:34:34.0627018Z f"transformers>={required}, but got {installed}") 2025-10-10T01:34:34.0627557Z  2025-10-10T01:34:34.0627806Z > check_implementation(hf_runner, 2025-10-10T01:34:34.0628105Z vllm_runner, 2025-10-10T01:34:34.0628388Z example_prompts, 2025-10-10T01:34:34.0628664Z model, 2025-10-10T01:34:34.0628950Z model_impl=model_impl) 2025-10-10T01:34:34.0629115Z 2025-10-10T01:34:34.0629255Z models/test_transformers.py:86: 2025-10-10T01:34:34.0629510Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0629883Z models/test_transformers.py:41: in check_implementation 2025-10-10T01:34:34.0630537Z with runner_test(model, **kwargs_test, **kwargs) as model_test: 2025-10-10T01:34:34.0630963Z conftest.py:738: in __init__ 2025-10-10T01:34:34.0631251Z self.llm = LLM( 2025-10-10T01:34:34.0631724Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.0632254Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.0632833Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.0633406Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.0633955Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.0634529Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.0635105Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.0635702Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.0636291Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.0636788Z super().__init__( 2025-10-10T01:34:34.0637292Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.0637854Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.0638330Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.0638746Z next(self.gen) 2025-10-10T01:34:34.0639417Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.0639896Z wait_for_engine_startup( 2025-10-10T01:34:34.0640159Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0640337Z 2025-10-10T01:34:34.0640485Z handshake_socket = 2025-10-10T01:34:34.0641319Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/221ad030-4e2a-428b-a5c4-53fd01107fd3'], outputs=['ipc:///tmp/1d34e04c-9a73-4d3c-825b-c859748633ad'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.0642170Z core_engines = [] 2025-10-10T01:34:34.0643041Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.0644396Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.0645236Z proc_manager = 2025-10-10T01:34:34.0645577Z coord_process = None 2025-10-10T01:34:34.0645685Z 2025-10-10T01:34:34.0664551Z def wait_for_engine_startup( 2025-10-10T01:34:34.0665161Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.0665822Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.0666461Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.0666937Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.0667491Z cache_config: CacheConfig, 2025-10-10T01:34:34.0668049Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.0668507Z coord_process: Optional[Process], 2025-10-10T01:34:34.0668832Z ): 2025-10-10T01:34:34.0669270Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.0669851Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.0670445Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.0670923Z # [local, remote] counts 2025-10-10T01:34:34.0671503Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.0671950Z poller = zmq.Poller() 2025-10-10T01:34:34.0672301Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.0672601Z  2025-10-10T01:34:34.0672955Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.0673483Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.0673836Z  2025-10-10T01:34:34.0674204Z if proc_manager is not None: 2025-10-10T01:34:34.0674712Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.0675122Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0675582Z if coord_process is not None: 2025-10-10T01:34:34.0676064Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0676613Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.0677098Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.0677474Z if not events: 2025-10-10T01:34:34.0677846Z if any(conn_pending): 2025-10-10T01:34:34.0678169Z logger.debug( 2025-10-10T01:34:34.0678759Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0679521Z "to connect.", *conn_pending) 2025-10-10T01:34:34.0680057Z if any(start_pending): 2025-10-10T01:34:34.0680386Z logger.debug( 2025-10-10T01:34:34.0680970Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0681584Z "to start.", *start_pending) 2025-10-10T01:34:34.0681950Z continue 2025-10-10T01:34:34.0682488Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.0683507Z # One of the local core processes exited. 2025-10-10T01:34:34.0684441Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.0685448Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.0686142Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.0686783Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.0687391Z "See root cause above. " 2025-10-10T01:34:34.0688139Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.0688916Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.0689373Z 2025-10-10T01:34:34.0689715Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.0690183Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.0690746Z WARNING  transformers.configuration_utils:logging.py:328 `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:34:34.0691774Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.0692637Z ____________________ test_models[hmellor/Ilama-3.2-1B-auto] ____________________ 2025-10-10T01:34:34.0692899Z 2025-10-10T01:34:34.0693001Z hf_runner = 2025-10-10T01:34:34.0693280Z vllm_runner = 2025-10-10T01:34:34.0694084Z example_prompts = ['vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n', 'Briefly describe the majo...me.\n', 'Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.\n', ...] 2025-10-10T01:34:34.0695020Z model = 'hmellor/Ilama-3.2-1B', model_impl = 'auto' 2025-10-10T01:34:34.0695209Z 2025-10-10T01:34:34.0695396Z @pytest.mark.skipif( 2025-10-10T01:34:34.0695709Z current_platform.is_rocm(), 2025-10-10T01:34:34.0696422Z reason="Llama-3.2-1B-Instruct, Ilama-3.2-1B produce memory access fault.") 2025-10-10T01:34:34.0697100Z @pytest.mark.parametrize( 2025-10-10T01:34:34.0697779Z "model,model_impl", 2025-10-10T01:34:34.0698261Z [ 2025-10-10T01:34:34.0698807Z ("meta-llama/Llama-3.2-1B-Instruct", "transformers"), 2025-10-10T01:34:34.0699572Z ("hmellor/Ilama-3.2-1B", "auto"), # CUSTOM CODE 2025-10-10T01:34:34.0700479Z ("allenai/OLMoE-1B-7B-0924", "transformers"), # MoE 2025-10-10T01:34:34.0701060Z ]) # trust_remote_code=True by default 2025-10-10T01:34:34.0701568Z def test_models( 2025-10-10T01:34:34.0701940Z hf_runner: type[HfRunner], 2025-10-10T01:34:34.0702533Z vllm_runner: type[VllmRunner], 2025-10-10T01:34:34.0703181Z example_prompts: list[str], 2025-10-10T01:34:34.0703723Z model: str, 2025-10-10T01:34:34.0704178Z model_impl: str, 2025-10-10T01:34:34.0704468Z ) -> None: 2025-10-10T01:34:34.0704805Z import transformers 2025-10-10T01:34:34.0705374Z from packaging.version import Version 2025-10-10T01:34:34.0705899Z installed = Version(transformers.__version__) 2025-10-10T01:34:34.0706344Z required = Version("4.57.0.dev0") 2025-10-10T01:34:34.0706998Z if model == "allenai/OLMoE-1B-7B-0924" and installed < required: 2025-10-10T01:34:34.0707718Z pytest.skip("MoE models with the Transformers backend require " 2025-10-10T01:34:34.0708555Z f"transformers>={required}, but got {installed}") 2025-10-10T01:34:34.0709084Z  2025-10-10T01:34:34.0709328Z > check_implementation(hf_runner, 2025-10-10T01:34:34.0709632Z vllm_runner, 2025-10-10T01:34:34.0709926Z example_prompts, 2025-10-10T01:34:34.0710212Z model, 2025-10-10T01:34:34.0710500Z model_impl=model_impl) 2025-10-10T01:34:34.0710673Z 2025-10-10T01:34:34.0710807Z models/test_transformers.py:86: 2025-10-10T01:34:34.0711067Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0711451Z models/test_transformers.py:41: in check_implementation 2025-10-10T01:34:34.0711966Z with runner_test(model, **kwargs_test, **kwargs) as model_test: 2025-10-10T01:34:34.0712378Z conftest.py:738: in __init__ 2025-10-10T01:34:34.0712665Z self.llm = LLM( 2025-10-10T01:34:34.0713148Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.0713676Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.0714382Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.0714956Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.0715532Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.0716092Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.0716670Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.0717284Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.0717885Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.0718408Z super().__init__( 2025-10-10T01:34:34.0719015Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.0719709Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.0720204Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.0720617Z next(self.gen) 2025-10-10T01:34:34.0721154Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.0721636Z wait_for_engine_startup( 2025-10-10T01:34:34.0721894Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0722066Z 2025-10-10T01:34:34.0722221Z handshake_socket = 2025-10-10T01:34:34.0723052Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/13d7d34a-1377-4e99-8103-38b48b210ab0'], outputs=['ipc:///tmp/667db6ae-416d-4a70-8404-3120f955ae90'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.0723917Z core_engines = [] 2025-10-10T01:34:34.0724777Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.0726040Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.0726894Z proc_manager = 2025-10-10T01:34:34.0727252Z coord_process = None 2025-10-10T01:34:34.0727367Z 2025-10-10T01:34:34.0735080Z def wait_for_engine_startup( 2025-10-10T01:34:34.0735763Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.0736418Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.0737001Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.0737460Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.0737849Z cache_config: CacheConfig, 2025-10-10T01:34:34.0738284Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.0738738Z coord_process: Optional[Process], 2025-10-10T01:34:34.0739066Z ): 2025-10-10T01:34:34.0739507Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.0740099Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.0740838Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.0741427Z # [local, remote] counts 2025-10-10T01:34:34.0742162Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.0742752Z poller = zmq.Poller() 2025-10-10T01:34:34.0743154Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.0743501Z  2025-10-10T01:34:34.0743862Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.0744389Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.0744734Z  2025-10-10T01:34:34.0745111Z if proc_manager is not None: 2025-10-10T01:34:34.0745846Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.0746494Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0747007Z if coord_process is not None: 2025-10-10T01:34:34.0747482Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0748028Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.0748518Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.0748885Z if not events: 2025-10-10T01:34:34.0749360Z if any(conn_pending): 2025-10-10T01:34:34.0749715Z logger.debug( 2025-10-10T01:34:34.0750311Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0751067Z "to connect.", *conn_pending) 2025-10-10T01:34:34.0751500Z if any(start_pending): 2025-10-10T01:34:34.0751831Z logger.debug( 2025-10-10T01:34:34.0752406Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0753146Z "to start.", *start_pending) 2025-10-10T01:34:34.0753529Z continue 2025-10-10T01:34:34.0754079Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.0754763Z # One of the local core processes exited. 2025-10-10T01:34:34.0755304Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.0756142Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.0756950Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.0757717Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.0758411Z "See root cause above. " 2025-10-10T01:34:34.0759192Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.0760038Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.0760423Z 2025-10-10T01:34:34.0760767Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.0761245Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.0762264Z WARNING  transformers.dynamic_module_utils:dynamic_module_utils.py:450 A new version of the following files was downloaded from https://huggingface.co/hmellor/Ilama-3.2-1B: 2025-10-10T01:34:34.0762884Z - configuration_ilama.py 2025-10-10T01:34:34.0763487Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:34:34.0764597Z WARNING  transformers.dynamic_module_utils:dynamic_module_utils.py:450 A new version of the following files was downloaded from https://huggingface.co/hmellor/Ilama-3.2-1B: 2025-10-10T01:34:34.0765477Z - modeling_ilama.py 2025-10-10T01:34:34.0765952Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:34:34.0766927Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.0767656Z ____________________________ test_hybrid_attention _____________________________ 2025-10-10T01:34:34.0767890Z 2025-10-10T01:34:34.0768005Z vllm_runner = 2025-10-10T01:34:34.0768183Z 2025-10-10T01:34:34.0768570Z def test_hybrid_attention(vllm_runner: type[VllmRunner]) -> None: 2025-10-10T01:34:34.0769201Z prompts, _, _ = prep_prompts(4, (800, 801)) 2025-10-10T01:34:34.0769931Z kwargs_ref = {"max_model_len": 8192, "enforce_eager": True} 2025-10-10T01:34:34.0770743Z kwargs_test = {"model_impl": "transformers", **kwargs_ref} 2025-10-10T01:34:34.0771256Z > check_implementation(vllm_runner, 2025-10-10T01:34:34.0771561Z vllm_runner, 2025-10-10T01:34:34.0771835Z prompts, 2025-10-10T01:34:34.0772338Z model="hmellor/tiny-random-Gemma2ForCausalLM", 2025-10-10T01:34:34.0772802Z kwargs_ref=kwargs_ref, 2025-10-10T01:34:34.0773117Z kwargs_test=kwargs_test) 2025-10-10T01:34:34.0773285Z 2025-10-10T01:34:34.0773420Z models/test_transformers.py:97: 2025-10-10T01:34:34.0773679Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0774049Z models/test_transformers.py:41: in check_implementation 2025-10-10T01:34:34.0774562Z with runner_test(model, **kwargs_test, **kwargs) as model_test: 2025-10-10T01:34:34.0775094Z conftest.py:738: in __init__ 2025-10-10T01:34:34.0775379Z self.llm = LLM( 2025-10-10T01:34:34.0775868Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.0776398Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.0776997Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.0777580Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.0778137Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.0778713Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.0779386Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.0780011Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.0780609Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.0781108Z super().__init__( 2025-10-10T01:34:34.0781630Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.0782211Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.0782685Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.0783097Z next(self.gen) 2025-10-10T01:34:34.0783628Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.0784110Z wait_for_engine_startup( 2025-10-10T01:34:34.0784377Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0784552Z 2025-10-10T01:34:34.0784703Z handshake_socket = 2025-10-10T01:34:34.0785537Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/55684a84-f671-439b-8088-b62dd57920c2'], outputs=['ipc:///tmp/60c6b913-6d59-4e32-97cc-a9f8645b1e33'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.0786368Z core_engines = [] 2025-10-10T01:34:34.0787225Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.0788508Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.0789364Z proc_manager = 2025-10-10T01:34:34.0789707Z coord_process = None 2025-10-10T01:34:34.0789814Z 2025-10-10T01:34:34.0797092Z def wait_for_engine_startup( 2025-10-10T01:34:34.0797959Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.0798558Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.0799363Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.0799864Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.0800626Z cache_config: CacheConfig, 2025-10-10T01:34:34.0801076Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.0801493Z coord_process: Optional[Process], 2025-10-10T01:34:34.0801775Z ): 2025-10-10T01:34:34.0802143Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.0802614Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.0803060Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.0803455Z # [local, remote] counts 2025-10-10T01:34:34.0803949Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.0804380Z poller = zmq.Poller() 2025-10-10T01:34:34.0804739Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.0805046Z  2025-10-10T01:34:34.0805530Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.0806086Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.0806449Z  2025-10-10T01:34:34.0806812Z if proc_manager is not None: 2025-10-10T01:34:34.0807318Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.0807740Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0808197Z if coord_process is not None: 2025-10-10T01:34:34.0808688Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0809234Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.0809722Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.0810094Z if not events: 2025-10-10T01:34:34.0810468Z if any(conn_pending): 2025-10-10T01:34:34.0810790Z logger.debug( 2025-10-10T01:34:34.0811401Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0812016Z "to connect.", *conn_pending) 2025-10-10T01:34:34.0812445Z if any(start_pending): 2025-10-10T01:34:34.0812777Z logger.debug( 2025-10-10T01:34:34.0813354Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0813946Z "to start.", *start_pending) 2025-10-10T01:34:34.0814310Z continue 2025-10-10T01:34:34.0815026Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.0816074Z # One of the local core processes exited. 2025-10-10T01:34:34.0817113Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.0818109Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.0818807Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.0819584Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.0820204Z "See root cause above. " 2025-10-10T01:34:34.0820940Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.0821796Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.0822188Z 2025-10-10T01:34:34.0822647Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.0823111Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.0823878Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.0824830Z _ test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ-quantization_kwargs0] _ 2025-10-10T01:34:34.0825156Z 2025-10-10T01:34:34.0825270Z vllm_runner = 2025-10-10T01:34:34.0826090Z example_prompts = ['vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n', 'Briefly describe the majo...me.\n', 'Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.\n', ...] 2025-10-10T01:34:34.0827066Z model = 'TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ', quantization_kwargs = {} 2025-10-10T01:34:34.0827443Z max_tokens = 32, num_logprobs = 5 2025-10-10T01:34:34.0827658Z 2025-10-10T01:34:34.0828049Z @pytest.mark.parametrize("model, quantization_kwargs", [ 2025-10-10T01:34:34.0828698Z ("TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ", {}), 2025-10-10T01:34:34.0829317Z ("TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ", {}), 2025-10-10T01:34:34.0829790Z ( 2025-10-10T01:34:34.0830172Z "meta-llama/Llama-3.2-1B-Instruct", 2025-10-10T01:34:34.0830663Z { 2025-10-10T01:34:34.0831162Z "quantization": "bitsandbytes", 2025-10-10T01:34:34.0831640Z }, 2025-10-10T01:34:34.0831955Z ), 2025-10-10T01:34:34.0832163Z ]) 2025-10-10T01:34:34.0832852Z @pytest.mark.parametrize("max_tokens", [32]) 2025-10-10T01:34:34.0833576Z @pytest.mark.parametrize("num_logprobs", [5]) 2025-10-10T01:34:34.0834106Z def test_quantization( 2025-10-10T01:34:34.0834492Z vllm_runner: type[VllmRunner], 2025-10-10T01:34:34.0834904Z example_prompts: list[str], 2025-10-10T01:34:34.0835390Z model: str, 2025-10-10T01:34:34.0835821Z quantization_kwargs: dict[str, str], 2025-10-10T01:34:34.0836236Z max_tokens: int, 2025-10-10T01:34:34.0836538Z num_logprobs: int, 2025-10-10T01:34:34.0836831Z ) -> None: 2025-10-10T01:34:34.0837164Z if (current_platform.is_rocm() 2025-10-10T01:34:34.0837901Z and quantization_kwargs.get("quantization", "") == "bitsandbytes"): 2025-10-10T01:34:34.0838486Z pytest.skip( 2025-10-10T01:34:34.0838993Z "bitsandbytes quantization is currently not supported in rocm.") 2025-10-10T01:34:34.0839716Z  2025-10-10T01:34:34.0839978Z > with vllm_runner( 2025-10-10T01:34:34.0840459Z model, model_impl="auto", enforce_eager=True, 2025-10-10T01:34:34.0841061Z **quantization_kwargs) as vllm_model: # type: ignore[arg-type] 2025-10-10T01:34:34.0841370Z 2025-10-10T01:34:34.0841502Z models/test_transformers.py:144: 2025-10-10T01:34:34.0841778Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0842077Z conftest.py:738: in __init__ 2025-10-10T01:34:34.0842365Z self.llm = LLM( 2025-10-10T01:34:34.0842843Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.0843397Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.0843984Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.0844572Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.0845130Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.0845682Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.0846246Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.0846838Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.0847446Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.0847954Z super().__init__( 2025-10-10T01:34:34.0848504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.0849075Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.0849547Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.0849967Z next(self.gen) 2025-10-10T01:34:34.0850506Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.0851062Z wait_for_engine_startup( 2025-10-10T01:34:34.0851365Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0851684Z 2025-10-10T01:34:34.0851865Z handshake_socket = 2025-10-10T01:34:34.0852876Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/4a14edd2-6436-4ed9-acba-951461263fd6'], outputs=['ipc:///tmp/b23e20fb-f42a-4318-82df-860e0fff8a7b'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.0853738Z core_engines = [] 2025-10-10T01:34:34.0854592Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.0855838Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.0856752Z proc_manager = 2025-10-10T01:34:34.0857102Z coord_process = None 2025-10-10T01:34:34.0857218Z 2025-10-10T01:34:34.0863818Z def wait_for_engine_startup( 2025-10-10T01:34:34.0864348Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.0864746Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.0865315Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.0865747Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.0866201Z cache_config: CacheConfig, 2025-10-10T01:34:34.0866790Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.0867175Z coord_process: Optional[Process], 2025-10-10T01:34:34.0867476Z ): 2025-10-10T01:34:34.0867843Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.0868329Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.0868775Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.0869156Z # [local, remote] counts 2025-10-10T01:34:34.0869654Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.0870088Z poller = zmq.Poller() 2025-10-10T01:34:34.0870436Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.0870741Z  2025-10-10T01:34:34.0871101Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.0871631Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.0871995Z  2025-10-10T01:34:34.0872365Z if proc_manager is not None: 2025-10-10T01:34:34.0872862Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.0873279Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0873739Z if coord_process is not None: 2025-10-10T01:34:34.0874226Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0874775Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.0875257Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.0875761Z if not events: 2025-10-10T01:34:34.0876139Z if any(conn_pending): 2025-10-10T01:34:34.0876463Z logger.debug( 2025-10-10T01:34:34.0877049Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0877656Z "to connect.", *conn_pending) 2025-10-10T01:34:34.0878082Z if any(start_pending): 2025-10-10T01:34:34.0878403Z logger.debug( 2025-10-10T01:34:34.0878976Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0879804Z "to start.", *start_pending) 2025-10-10T01:34:34.0880170Z continue 2025-10-10T01:34:34.0880728Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.0881287Z # One of the local core processes exited. 2025-10-10T01:34:34.0881830Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.0882673Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.0883736Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.0884744Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.0885401Z "See root cause above. " 2025-10-10T01:34:34.0886014Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.0886880Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.0887265Z 2025-10-10T01:34:34.0887617Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.0888213Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.0888987Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.0889936Z _ test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ-quantization_kwargs1] _ 2025-10-10T01:34:34.0890272Z 2025-10-10T01:34:34.0890385Z vllm_runner = 2025-10-10T01:34:34.0891214Z example_prompts = ['vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n', 'Briefly describe the majo...me.\n', 'Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.\n', ...] 2025-10-10T01:34:34.0892211Z model = 'TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ', quantization_kwargs = {} 2025-10-10T01:34:34.0892672Z max_tokens = 32, num_logprobs = 5 2025-10-10T01:34:34.0892814Z 2025-10-10T01:34:34.0893242Z @pytest.mark.parametrize("model, quantization_kwargs", [ 2025-10-10T01:34:34.0894053Z ("TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ", {}), 2025-10-10T01:34:34.0894643Z ("TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ", {}), 2025-10-10T01:34:34.0895178Z ( 2025-10-10T01:34:34.0895573Z "meta-llama/Llama-3.2-1B-Instruct", 2025-10-10T01:34:34.0895938Z { 2025-10-10T01:34:34.0896615Z "quantization": "bitsandbytes", 2025-10-10T01:34:34.0897185Z }, 2025-10-10T01:34:34.0897421Z ), 2025-10-10T01:34:34.0897622Z ]) 2025-10-10T01:34:34.0898313Z @pytest.mark.parametrize("max_tokens", [32]) 2025-10-10T01:34:34.0899232Z @pytest.mark.parametrize("num_logprobs", [5]) 2025-10-10T01:34:34.0899946Z def test_quantization( 2025-10-10T01:34:34.0900424Z vllm_runner: type[VllmRunner], 2025-10-10T01:34:34.0900855Z example_prompts: list[str], 2025-10-10T01:34:34.0901206Z model: str, 2025-10-10T01:34:34.0911944Z quantization_kwargs: dict[str, str], 2025-10-10T01:34:34.0912427Z max_tokens: int, 2025-10-10T01:34:34.0912760Z num_logprobs: int, 2025-10-10T01:34:34.0913083Z ) -> None: 2025-10-10T01:34:34.0913405Z if (current_platform.is_rocm() 2025-10-10T01:34:34.0914149Z and quantization_kwargs.get("quantization", "") == "bitsandbytes"): 2025-10-10T01:34:34.0914737Z pytest.skip( 2025-10-10T01:34:34.0915266Z "bitsandbytes quantization is currently not supported in rocm.") 2025-10-10T01:34:34.0915714Z  2025-10-10T01:34:34.0915958Z > with vllm_runner( 2025-10-10T01:34:34.0916440Z model, model_impl="auto", enforce_eager=True, 2025-10-10T01:34:34.0917057Z **quantization_kwargs) as vllm_model: # type: ignore[arg-type] 2025-10-10T01:34:34.0917360Z 2025-10-10T01:34:34.0917512Z models/test_transformers.py:144: 2025-10-10T01:34:34.0917806Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0918118Z conftest.py:738: in __init__ 2025-10-10T01:34:34.0918422Z self.llm = LLM( 2025-10-10T01:34:34.0918919Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.0919592Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.0920197Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.0920996Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.0921571Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.0922134Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.0922714Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.0923345Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.0923943Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.0924452Z super().__init__( 2025-10-10T01:34:34.0924969Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.0925651Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.0926142Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.0926565Z next(self.gen) 2025-10-10T01:34:34.0927104Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.0927603Z wait_for_engine_startup( 2025-10-10T01:34:34.0927862Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0928037Z 2025-10-10T01:34:34.0928192Z handshake_socket = 2025-10-10T01:34:34.0929309Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/313863a9-8c36-401d-b455-1fd66085ad0c'], outputs=['ipc:///tmp/62d3c234-6ca0-481f-8464-b38f9c381e13'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.0930620Z core_engines = [] 2025-10-10T01:34:34.0932074Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.0933508Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.0934356Z proc_manager = 2025-10-10T01:34:34.0934711Z coord_process = None 2025-10-10T01:34:34.0934830Z 2025-10-10T01:34:34.0935092Z def wait_for_engine_startup( 2025-10-10T01:34:34.0935470Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.0935796Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.0936156Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.0936515Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.0936825Z cache_config: CacheConfig, 2025-10-10T01:34:34.0937181Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.0937543Z coord_process: Optional[Process], 2025-10-10T01:34:34.0937813Z ): 2025-10-10T01:34:34.0938162Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.0938648Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.0939093Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.0939617Z # [local, remote] counts 2025-10-10T01:34:34.0940115Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.0940557Z poller = zmq.Poller() 2025-10-10T01:34:34.0940911Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.0941221Z  2025-10-10T01:34:34.0941586Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.0942113Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.0942482Z  2025-10-10T01:34:34.0942852Z if proc_manager is not None: 2025-10-10T01:34:34.0943361Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.0943866Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0944339Z if coord_process is not None: 2025-10-10T01:34:34.0944830Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.0945370Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.0945854Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.0946219Z if not events: 2025-10-10T01:34:34.0946582Z if any(conn_pending): 2025-10-10T01:34:34.0946904Z logger.debug( 2025-10-10T01:34:34.0947947Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0948964Z "to connect.", *conn_pending) 2025-10-10T01:34:34.0949735Z if any(start_pending): 2025-10-10T01:34:34.0950080Z logger.debug( 2025-10-10T01:34:34.0950699Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.0951416Z "to start.", *start_pending) 2025-10-10T01:34:34.0951786Z continue 2025-10-10T01:34:34.0952584Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.0953302Z # One of the local core processes exited. 2025-10-10T01:34:34.0953948Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.0955018Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.0955793Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.0956629Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.0957585Z "See root cause above. " 2025-10-10T01:34:34.0958425Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.0959501Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.0960141Z 2025-10-10T01:34:34.0960759Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.0961261Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.0962035Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.0962952Z _ test_quantization[5-32-meta-llama/Llama-3.2-1B-Instruct-quantization_kwargs2] _ 2025-10-10T01:34:34.0963248Z 2025-10-10T01:34:34.0963360Z vllm_runner = 2025-10-10T01:34:34.0964174Z example_prompts = ['vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n', 'Briefly describe the majo...me.\n', 'Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.\n', ...] 2025-10-10T01:34:34.0964971Z model = 'meta-llama/Llama-3.2-1B-Instruct' 2025-10-10T01:34:34.0965297Z quantization_kwargs = {'quantization': 'bitsandbytes'}, max_tokens = 32 2025-10-10T01:34:34.0965601Z num_logprobs = 5 2025-10-10T01:34:34.0965707Z 2025-10-10T01:34:34.0966109Z @pytest.mark.parametrize("model, quantization_kwargs", [ 2025-10-10T01:34:34.0966756Z ("TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ", {}), 2025-10-10T01:34:34.0967339Z ("TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ", {}), 2025-10-10T01:34:34.0967727Z ( 2025-10-10T01:34:34.0968115Z "meta-llama/Llama-3.2-1B-Instruct", 2025-10-10T01:34:34.0968477Z { 2025-10-10T01:34:34.0968941Z "quantization": "bitsandbytes", 2025-10-10T01:34:34.0969372Z }, 2025-10-10T01:34:34.0969587Z ), 2025-10-10T01:34:34.0969791Z ]) 2025-10-10T01:34:34.0970238Z @pytest.mark.parametrize("max_tokens", [32]) 2025-10-10T01:34:34.0970932Z @pytest.mark.parametrize("num_logprobs", [5]) 2025-10-10T01:34:34.0971450Z def test_quantization( 2025-10-10T01:34:34.0971835Z vllm_runner: type[VllmRunner], 2025-10-10T01:34:34.0972248Z example_prompts: list[str], 2025-10-10T01:34:34.0972607Z model: str, 2025-10-10T01:34:34.0973028Z quantization_kwargs: dict[str, str], 2025-10-10T01:34:34.0973440Z max_tokens: int, 2025-10-10T01:34:34.0973738Z num_logprobs: int, 2025-10-10T01:34:34.0974028Z ) -> None: 2025-10-10T01:34:34.0974451Z if (current_platform.is_rocm() 2025-10-10T01:34:34.0975187Z and quantization_kwargs.get("quantization", "") == "bitsandbytes"): 2025-10-10T01:34:34.0975765Z pytest.skip( 2025-10-10T01:34:34.0976264Z "bitsandbytes quantization is currently not supported in rocm.") 2025-10-10T01:34:34.0976692Z  2025-10-10T01:34:34.0976928Z > with vllm_runner( 2025-10-10T01:34:34.0977400Z model, model_impl="auto", enforce_eager=True, 2025-10-10T01:34:34.0978008Z **quantization_kwargs) as vllm_model: # type: ignore[arg-type] 2025-10-10T01:34:34.0978309Z 2025-10-10T01:34:34.0978441Z models/test_transformers.py:144: 2025-10-10T01:34:34.0978788Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0979092Z conftest.py:738: in __init__ 2025-10-10T01:34:34.0979394Z self.llm = LLM( 2025-10-10T01:34:34.0979880Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.0980399Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.0980995Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.0981566Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.0982116Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.0982682Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.0983252Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.0983848Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.0984440Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.0984943Z super().__init__( 2025-10-10T01:34:34.0985458Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.0986015Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.0986489Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.0986902Z next(self.gen) 2025-10-10T01:34:34.0987452Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.0988018Z wait_for_engine_startup( 2025-10-10T01:34:34.0988485Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.0988816Z 2025-10-10T01:34:34.0989083Z handshake_socket = 2025-10-10T01:34:34.0990627Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/62cb4c69-8281-4f35-94a3-5b6e4ae9c8ed'], outputs=['ipc:///tmp/6fc799cf-ebd6-489a-a667-e1ad3ca6b429'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.0991737Z core_engines = [] 2025-10-10T01:34:34.0992619Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.0994013Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.0994869Z proc_manager = 2025-10-10T01:34:34.0995217Z coord_process = None 2025-10-10T01:34:34.0995345Z 2025-10-10T01:34:34.0995597Z def wait_for_engine_startup( 2025-10-10T01:34:34.0995961Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.0996478Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.0996839Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.0997421Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.0998195Z cache_config: CacheConfig, 2025-10-10T01:34:34.0998700Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.0999343Z coord_process: Optional[Process], 2025-10-10T01:34:34.0999729Z ): 2025-10-10T01:34:34.1000230Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.1000724Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.1001413Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.1001828Z # [local, remote] counts 2025-10-10T01:34:34.1002557Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.1003033Z poller = zmq.Poller() 2025-10-10T01:34:34.1003550Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.1003917Z  2025-10-10T01:34:34.1004286Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.1005033Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.1005422Z  2025-10-10T01:34:34.1006009Z if proc_manager is not None: 2025-10-10T01:34:34.1006543Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.1007185Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.1007713Z if coord_process is not None: 2025-10-10T01:34:34.1008468Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.1009448Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.1009986Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.1010359Z if not events: 2025-10-10T01:34:34.1010736Z if any(conn_pending): 2025-10-10T01:34:34.1011067Z logger.debug( 2025-10-10T01:34:34.1011664Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.1012276Z "to connect.", *conn_pending) 2025-10-10T01:34:34.1012879Z if any(start_pending): 2025-10-10T01:34:34.1013200Z logger.debug( 2025-10-10T01:34:34.1013780Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.1014377Z "to start.", *start_pending) 2025-10-10T01:34:34.1014733Z continue 2025-10-10T01:34:34.1015270Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.1015814Z # One of the local core processes exited. 2025-10-10T01:34:34.1016451Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.1017308Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.1018001Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.1018639Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.1019254Z "See root cause above. " 2025-10-10T01:34:34.1019850Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.1020570Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1020956Z 2025-10-10T01:34:34.1021289Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.1021753Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.1022491Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.1023256Z ________________ test_embed_loading[Qwen/Qwen3-Embedding-0.6B] _________________ 2025-10-10T01:34:34.1023519Z 2025-10-10T01:34:34.1023627Z vllm_runner = 2025-10-10T01:34:34.1023891Z model = 'Qwen/Qwen3-Embedding-0.6B' 2025-10-10T01:34:34.1024037Z 2025-10-10T01:34:34.1024233Z @pytest.mark.parametrize( 2025-10-10T01:34:34.1024609Z "model", 2025-10-10T01:34:34.1024893Z [ 2025-10-10T01:34:34.1025165Z # Layers live in `layers` 2025-10-10T01:34:34.1025595Z "Qwen/Qwen3-Embedding-0.6B", 2025-10-10T01:34:34.1026020Z # Layers live in `model.layers` 2025-10-10T01:34:34.1026467Z "meta-llama/Llama-3.2-1B-Instruct" 2025-10-10T01:34:34.1026826Z ], 2025-10-10T01:34:34.1027040Z ) 2025-10-10T01:34:34.1027384Z def test_embed_loading(vllm_runner, model): 2025-10-10T01:34:34.1027883Z > with vllm_runner(model, 2025-10-10T01:34:34.1028222Z max_model_len=1024, 2025-10-10T01:34:34.1028557Z enforce_eager=True, 2025-10-10T01:34:34.1028948Z runner="pooling", 2025-10-10T01:34:34.1029479Z model_impl="transformers") as model_test: 2025-10-10T01:34:34.1029776Z 2025-10-10T01:34:34.1029909Z models/test_transformers.py:179: 2025-10-10T01:34:34.1030171Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.1030467Z conftest.py:738: in __init__ 2025-10-10T01:34:34.1030808Z self.llm = LLM( 2025-10-10T01:34:34.1031491Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.1032111Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.1032749Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.1033322Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.1033876Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.1034429Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.1034999Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.1035592Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.1036200Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.1036701Z super().__init__( 2025-10-10T01:34:34.1037372Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.1038222Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.1038939Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.1039646Z next(self.gen) 2025-10-10T01:34:34.1040475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.1041115Z wait_for_engine_startup( 2025-10-10T01:34:34.1041388Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.1041570Z 2025-10-10T01:34:34.1041725Z handshake_socket = 2025-10-10T01:34:34.1042571Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/9147df94-b30d-4d96-a0de-e4458309f56b'], outputs=['ipc:///tmp/ae2fd3ca-f26b-4a69-b0a7-76dce75aa8a9'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.1043431Z core_engines = [] 2025-10-10T01:34:34.1044296Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.1045563Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.1046826Z proc_manager = 2025-10-10T01:34:34.1047418Z coord_process = None 2025-10-10T01:34:34.1047552Z 2025-10-10T01:34:34.1047889Z def wait_for_engine_startup( 2025-10-10T01:34:34.1048274Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.1048593Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.1049091Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.1049448Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.1049762Z cache_config: CacheConfig, 2025-10-10T01:34:34.1050238Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.1050614Z coord_process: Optional[Process], 2025-10-10T01:34:34.1050895Z ): 2025-10-10T01:34:34.1051480Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.1051992Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.1052556Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.1052970Z # [local, remote] counts 2025-10-10T01:34:34.1053456Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.1054030Z poller = zmq.Poller() 2025-10-10T01:34:34.1054373Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.1054770Z  2025-10-10T01:34:34.1055136Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.1055707Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.1056180Z  2025-10-10T01:34:34.1056554Z if proc_manager is not None: 2025-10-10T01:34:34.1057180Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.1057754Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.1058352Z if coord_process is not None: 2025-10-10T01:34:34.1058838Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.1059392Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.1059880Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.1060252Z if not events: 2025-10-10T01:34:34.1060619Z if any(conn_pending): 2025-10-10T01:34:34.1060939Z logger.debug( 2025-10-10T01:34:34.1061517Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.1062131Z "to connect.", *conn_pending) 2025-10-10T01:34:34.1062562Z if any(start_pending): 2025-10-10T01:34:34.1062882Z logger.debug( 2025-10-10T01:34:34.1063457Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.1064191Z "to start.", *start_pending) 2025-10-10T01:34:34.1064549Z continue 2025-10-10T01:34:34.1065105Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.1065653Z # One of the local core processes exited. 2025-10-10T01:34:34.1066191Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.1067027Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.1067780Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.1068432Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.1069045Z "See root cause above. " 2025-10-10T01:34:34.1069640Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.1070343Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1070715Z 2025-10-10T01:34:34.1071047Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.1071500Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.1072248Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.1072992Z _____________ test_embed_loading[meta-llama/Llama-3.2-1B-Instruct] _____________ 2025-10-10T01:34:34.1073253Z 2025-10-10T01:34:34.1073360Z vllm_runner = 2025-10-10T01:34:34.1073624Z model = 'meta-llama/Llama-3.2-1B-Instruct' 2025-10-10T01:34:34.1073780Z 2025-10-10T01:34:34.1073954Z @pytest.mark.parametrize( 2025-10-10T01:34:34.1074328Z "model", 2025-10-10T01:34:34.1074616Z [ 2025-10-10T01:34:34.1074889Z # Layers live in `layers` 2025-10-10T01:34:34.1075320Z "Qwen/Qwen3-Embedding-0.6B", 2025-10-10T01:34:34.1075739Z # Layers live in `model.layers` 2025-10-10T01:34:34.1076193Z "meta-llama/Llama-3.2-1B-Instruct" 2025-10-10T01:34:34.1076548Z ], 2025-10-10T01:34:34.1076753Z ) 2025-10-10T01:34:34.1077089Z def test_embed_loading(vllm_runner, model): 2025-10-10T01:34:34.1077502Z > with vllm_runner(model, 2025-10-10T01:34:34.1077849Z max_model_len=1024, 2025-10-10T01:34:34.1078186Z enforce_eager=True, 2025-10-10T01:34:34.1078674Z runner="pooling", 2025-10-10T01:34:34.1079339Z model_impl="transformers") as model_test: 2025-10-10T01:34:34.1079638Z 2025-10-10T01:34:34.1079775Z models/test_transformers.py:179: 2025-10-10T01:34:34.1080036Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.1080329Z conftest.py:738: in __init__ 2025-10-10T01:34:34.1080618Z self.llm = LLM( 2025-10-10T01:34:34.1081096Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T01:34:34.1081620Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T01:34:34.1082218Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T01:34:34.1083029Z return cls(vllm_config=vllm_config, 2025-10-10T01:34:34.1083606Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T01:34:34.1084150Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T01:34:34.1084721Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T01:34:34.1085314Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T01:34:34.1085904Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T01:34:34.1086741Z super().__init__( 2025-10-10T01:34:34.1087596Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T01:34:34.1088546Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T01:34:34.1089109Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T01:34:34.1089550Z next(self.gen) 2025-10-10T01:34:34.1090241Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T01:34:34.1090733Z wait_for_engine_startup( 2025-10-10T01:34:34.1090991Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T01:34:34.1091168Z 2025-10-10T01:34:34.1091314Z handshake_socket = 2025-10-10T01:34:34.1092295Z addresses = EngineZmqAddresses(inputs=['ipc:///tmp/5d9d92b3-53b2-4c9c-bf52-949f5cb5e43c'], outputs=['ipc:///tmp/eb1cf683-5936-48c3-bc6e-2101919d298e'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None) 2025-10-10T01:34:34.1093161Z core_engines = [] 2025-10-10T01:34:34.1094251Z parallel_config = ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=1, dat... rank=0, _data_parallel_master_port_list=[], decode_context_parallel_size=1, _api_process_count=1, _api_process_rank=0) 2025-10-10T01:34:34.1095517Z cache_config = CacheConfig(block_size=16, gpu_memory_utilization=0.9, swap_space=4.0, cache_dtype='auto', is_attention_free=False, nu...ache_dtype='auto', num_gpu_blocks=None, num_cpu_blocks=None, kv_sharing_fast_prefill=False, kv_cache_memory_bytes=None) 2025-10-10T01:34:34.1096544Z proc_manager = 2025-10-10T01:34:34.1096876Z coord_process = None 2025-10-10T01:34:34.1096984Z 2025-10-10T01:34:34.1097226Z def wait_for_engine_startup( 2025-10-10T01:34:34.1097848Z handshake_socket: zmq.Socket, 2025-10-10T01:34:34.1098163Z addresses: EngineZmqAddresses, 2025-10-10T01:34:34.1098511Z core_engines: list[CoreEngine], 2025-10-10T01:34:34.1098860Z parallel_config: ParallelConfig, 2025-10-10T01:34:34.1099166Z cache_config: CacheConfig, 2025-10-10T01:34:34.1099513Z proc_manager: Optional[CoreEngineProcManager], 2025-10-10T01:34:34.1099864Z coord_process: Optional[Process], 2025-10-10T01:34:34.1100118Z ): 2025-10-10T01:34:34.1100460Z # Wait for engine core process(es) to send ready messages. 2025-10-10T01:34:34.1100921Z local_count = parallel_config.data_parallel_size_local 2025-10-10T01:34:34.1101351Z remote_count = len(core_engines) - local_count 2025-10-10T01:34:34.1101839Z # [local, remote] counts 2025-10-10T01:34:34.1102340Z conn_pending, start_pending = [local_count, remote_count], [0, 0] 2025-10-10T01:34:34.1102768Z poller = zmq.Poller() 2025-10-10T01:34:34.1103126Z poller.register(handshake_socket, zmq.POLLIN) 2025-10-10T01:34:34.1103520Z  2025-10-10T01:34:34.1104087Z remote_should_be_headless = not parallel_config.data_parallel_hybrid_lb \ 2025-10-10T01:34:34.1104641Z and not parallel_config.data_parallel_external_lb 2025-10-10T01:34:34.1104994Z  2025-10-10T01:34:34.1105483Z if proc_manager is not None: 2025-10-10T01:34:34.1106032Z for sentinel in proc_manager.sentinels(): 2025-10-10T01:34:34.1106465Z poller.register(sentinel, zmq.POLLIN) 2025-10-10T01:34:34.1106930Z if coord_process is not None: 2025-10-10T01:34:34.1107412Z poller.register(coord_process.sentinel, zmq.POLLIN) 2025-10-10T01:34:34.1107952Z while any(conn_pending) or any(start_pending): 2025-10-10T01:34:34.1108437Z events = poller.poll(STARTUP_POLL_PERIOD_MS) 2025-10-10T01:34:34.1108804Z if not events: 2025-10-10T01:34:34.1109173Z if any(conn_pending): 2025-10-10T01:34:34.1109501Z logger.debug( 2025-10-10T01:34:34.1110099Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.1110706Z "to connect.", *conn_pending) 2025-10-10T01:34:34.1111130Z if any(start_pending): 2025-10-10T01:34:34.1111458Z logger.debug( 2025-10-10T01:34:34.1112038Z "Waiting for %d local, %d remote core engine proc(s) " 2025-10-10T01:34:34.1112641Z "to start.", *start_pending) 2025-10-10T01:34:34.1113006Z continue 2025-10-10T01:34:34.1113662Z if len(events) > 1 or events[0][0] != handshake_socket: 2025-10-10T01:34:34.1114207Z # One of the local core processes exited. 2025-10-10T01:34:34.1114748Z finished = proc_manager.finished_procs() if proc_manager else {} 2025-10-10T01:34:34.1115595Z if coord_process is not None and coord_process.exitcode is not None: 2025-10-10T01:34:34.1116277Z finished[coord_process.name] = coord_process.exitcode 2025-10-10T01:34:34.1116927Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T01:34:34.1117647Z "See root cause above. " 2025-10-10T01:34:34.1118260Z f"Failed core proc(s): {finished}") 2025-10-10T01:34:34.1118973Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1119463Z 2025-10-10T01:34:34.1119802Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T01:34:34.1120267Z ------------------------------ Captured log call ------------------------------- 2025-10-10T01:34:34.1121016Z WARNING  transformers.configuration_utils:configuration_utils.py:697 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:34:34.1121712Z =============================== warnings summary =============================== 2025-10-10T01:34:34.1122094Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64 2025-10-10T01:34:34.1123097Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:34:34.1123992Z import pynvml # type: ignore[import] 2025-10-10T01:34:34.1124155Z 2025-10-10T01:34:34.1124409Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305 2025-10-10T01:34:34.1125551Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable. 2025-10-10T01:34:34.1126605Z ref_error: type[Exception] = jsonschema.RefResolutionError, 2025-10-10T01:34:34.1126813Z 2025-10-10T01:34:34.1127067Z tests/models/test_transformers.py::test_models[meta-llama/Llama-3.2-1B-Instruct-transformers] 2025-10-10T01:34:34.1127541Z tests/models/test_transformers.py::test_models[hmellor/Ilama-3.2-1B-auto] 2025-10-10T01:34:34.1127911Z tests/models/test_transformers.py::test_hybrid_attention 2025-10-10T01:34:34.1128410Z tests/models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ-quantization_kwargs0] 2025-10-10T01:34:34.1129082Z tests/models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ-quantization_kwargs1] 2025-10-10T01:34:34.1129727Z tests/models/test_transformers.py::test_quantization[5-32-meta-llama/Llama-3.2-1B-Instruct-quantization_kwargs2] 2025-10-10T01:34:34.1130351Z tests/models/test_transformers.py::test_embed_loading[Qwen/Qwen3-Embedding-0.6B] 2025-10-10T01:34:34.1130813Z tests/models/test_transformers.py::test_embed_loading[meta-llama/Llama-3.2-1B-Instruct] 2025-10-10T01:34:34.1131554Z /opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=631) is multi-threaded, use of fork() may lead to deadlocks in the child. 2025-10-10T01:34:34.1132165Z self.pid = os.fork() 2025-10-10T01:34:34.1132279Z 2025-10-10T01:34:34.1132448Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2025-10-10T01:34:34.1132908Z =========================== short test summary info ============================ 2025-10-10T01:34:34.1136347Z FAILED models/test_transformers.py::test_models[meta-llama/Llama-3.2-1B-Instruct-transformers] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1137831Z FAILED models/test_transformers.py::test_models[hmellor/Ilama-3.2-1B-auto] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1139033Z FAILED models/test_transformers.py::test_hybrid_attention - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1140447Z FAILED models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ-quantization_kwargs0] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1141955Z FAILED models/test_transformers.py::test_quantization[5-32-TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ-quantization_kwargs1] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1143453Z FAILED models/test_transformers.py::test_quantization[5-32-meta-llama/Llama-3.2-1B-Instruct-quantization_kwargs2] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1144812Z FAILED models/test_transformers.py::test_embed_loading[Qwen/Qwen3-Embedding-0.6B] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1146077Z FAILED models/test_transformers.py::test_embed_loading[meta-llama/Llama-3.2-1B-Instruct] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T01:34:34.1146989Z ================== 8 failed, 4 skipped, 10 warnings in 40.98s ================== 2025-10-10T01:34:35.7887108Z 2025-10-10 01:34:35,788 [INFO] cli.lib.core.vllm.lib: Finish running step: pytest -v -s models/test_transformers.py 2025-10-10T01:34:35.7887906Z 2025-10-10 01:34:35,788 [INFO] cli.lib.core.vllm.lib: Running step: pytest -v -s models/test_registry.py 2025-10-10T01:34:35.7888515Z 2025-10-10 01:34:35,788 [INFO] cli.lib.common.utils: [shell] pytest -v -s models/test_registry.py 2025-10-10T01:34:40.4962118Z INFO 10-10 01:34:40 [__init__.py:215] Automatically detected platform cuda. 2025-10-10T01:34:42.1113885Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset. 2025-10-10T01:34:42.1115851Z The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session" 2025-10-10T01:34:42.1117508Z 2025-10-10T01:34:42.1117768Z warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) 2025-10-10T01:34:42.1180713Z ============================= test session starts ============================== 2025-10-10T01:34:42.1181276Z platform linux -- Python 3.12.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/conda/envs/py_3.12/bin/python 2025-10-10T01:34:42.1276114Z cachedir: .pytest_cache 2025-10-10T01:34:42.1276809Z hypothesis profile 'ci' -> database=None, deadline=None, print_blob=True, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-10-10T01:34:42.1277497Z rootdir: /var/lib/jenkins/workspace/vllm 2025-10-10T01:34:42.1277803Z configfile: pyproject.toml 2025-10-10T01:34:42.1278795Z plugins: cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, xdist-3.3.1, xdoctest-1.3.0, asyncio-0.24.0, cov-6.3.0, buildkite-test-collector-0.1.9, schemathesis-3.39.15, anyio-4.6.2.post1, hydra-core-1.3.2, mock-3.14.0, subtests-0.14.1, timeout-2.3.1, hypothesis-6.131.0, shard-0.1.2, forked-1.6.0 2025-10-10T01:34:42.1279744Z asyncio: mode=Mode.STRICT, default_loop_scope=None 2025-10-10T01:34:43.8697156Z collecting ... WARNING 10-10 01:34:43 [interface.py:488] Current platform cuda does not have '_pytestfixturefunction' attribute. 2025-10-10T01:34:44.1864200Z WARNING 10-10 01:34:44 [interface.py:488] Current platform cuda does not have '__test__' attribute. 2025-10-10T01:34:44.1864874Z WARNING 10-10 01:34:44 [interface.py:488] Current platform cuda does not have '__bases__' attribute. 2025-10-10T01:34:44.1865476Z WARNING 10-10 01:34:44 [interface.py:488] Current platform cuda does not have '__test__' attribute. 2025-10-10T01:34:44.1866125Z WARNING 10-10 01:34:44 [interface.py:488] Current platform cuda does not have '_schemathesis_test' attribute. 2025-10-10T01:34:44.1930164Z  2025-10-10T01:34:44.1954831Z collecting 226 items  2025-10-10T01:34:44.1955378Z collected 226 items  2025-10-10T01:34:44.2007400Z Running 226 items in this shard: tests/models/test_registry.py::test_registry_imports[ApertusForCausalLM], tests/models/test_registry.py::test_registry_imports[AquilaModel], tests/models/test_registry.py::test_registry_imports[AquilaForCausalLM], tests/models/test_registry.py::test_registry_imports[ArceeForCausalLM], tests/models/test_registry.py::test_registry_imports[ArcticForCausalLM], tests/models/test_registry.py::test_registry_imports[MiniMaxForCausalLM], tests/models/test_registry.py::test_registry_imports[MiniMaxText01ForCausalLM], tests/models/test_registry.py::test_registry_imports[MiniMaxM1ForCausalLM], tests/models/test_registry.py::test_registry_imports[BaiChuanForCausalLM], tests/models/test_registry.py::test_registry_imports[BaichuanForCausalLM], tests/models/test_registry.py::test_registry_imports[BailingMoeForCausalLM], tests/models/test_registry.py::test_registry_imports[BailingMoeV2ForCausalLM], tests/models/test_registry.py::test_registry_imports[BambaForCausalLM], tests/models/test_registry.py::test_registry_imports[BloomForCausalLM], tests/models/test_registry.py::test_registry_imports[ChatGLMModel], tests/models/test_registry.py::test_registry_imports[ChatGLMForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[CohereForCausalLM], tests/models/test_registry.py::test_registry_imports[Cohere2ForCausalLM], tests/models/test_registry.py::test_registry_imports[CwmForCausalLM], tests/models/test_registry.py::test_registry_imports[DbrxForCausalLM], tests/models/test_registry.py::test_registry_imports[DeciLMForCausalLM], tests/models/test_registry.py::test_registry_imports[DeepseekForCausalLM], tests/models/test_registry.py::test_registry_imports[DeepseekV2ForCausalLM], tests/models/test_registry.py::test_registry_imports[DeepseekV3ForCausalLM], tests/models/test_registry.py::test_registry_imports[DeepseekV32ForCausalLM], tests/models/test_registry.py::test_registry_imports[Dots1ForCausalLM], tests/models/test_registry.py::test_registry_imports[Ernie4_5ForCausalLM], tests/models/test_registry.py::test_registry_imports[Ernie4_5_MoeForCausalLM], tests/models/test_registry.py::test_registry_imports[ExaoneForCausalLM], tests/models/test_registry.py::test_registry_imports[Exaone4ForCausalLM], tests/models/test_registry.py::test_registry_imports[FalconForCausalLM], tests/models/test_registry.py::test_registry_imports[Fairseq2LlamaForCausalLM], tests/models/test_registry.py::test_registry_imports[GemmaForCausalLM], tests/models/test_registry.py::test_registry_imports[Gemma2ForCausalLM], tests/models/test_registry.py::test_registry_imports[Gemma3ForCausalLM], tests/models/test_registry.py::test_registry_imports[Gemma3nForCausalLM], tests/models/test_registry.py::test_registry_imports[Qwen3NextForCausalLM], tests/models/test_registry.py::test_registry_imports[GlmForCausalLM], tests/models/test_registry.py::test_registry_imports[Glm4ForCausalLM], tests/models/test_registry.py::test_registry_imports[Glm4MoeForCausalLM], tests/models/test_registry.py::test_registry_imports[GptOssForCausalLM], tests/models/test_registry.py::test_registry_imports[GPT2LMHeadModel], tests/models/test_registry.py::test_registry_imports[GPTBigCodeForCausalLM], tests/models/test_registry.py::test_registry_imports[GPTJForCausalLM], tests/models/test_registry.py::test_registry_imports[GPTNeoXForCausalLM], tests/models/test_registry.py::test_registry_imports[GraniteForCausalLM], tests/models/test_registry.py::test_registry_imports[GraniteMoeForCausalLM], tests/models/test_registry.py::test_registry_imports[GraniteMoeHybridForCausalLM], tests/models/test_registry.py::test_registry_imports[GraniteMoeSharedForCausalLM], tests/models/test_registry.py::test_registry_imports[GritLM], tests/models/test_registry.py::test_registry_imports[Grok1ModelForCausalLM], tests/models/test_registry.py::test_registry_imports[HunYuanMoEV1ForCausalLM], tests/models/test_registry.py::test_registry_imports[HunYuanDenseV1ForCausalLM], tests/models/test_registry.py::test_registry_imports[HCXVisionForCausalLM], tests/models/test_registry.py::test_registry_imports[InternLMForCausalLM], tests/models/test_registry.py::test_registry_imports[InternLM2ForCausalLM], tests/models/test_registry.py::test_registry_imports[InternLM2VEForCausalLM], tests/models/test_registry.py::test_registry_imports[InternLM3ForCausalLM], tests/models/test_registry.py::test_registry_imports[JAISLMHeadModel], tests/models/test_registry.py::test_registry_imports[JambaForCausalLM], tests/models/test_registry.py::test_registry_imports[Lfm2ForCausalLM], tests/models/test_registry.py::test_registry_imports[LlamaForCausalLM], tests/models/test_registry.py::test_registry_imports[Llama4ForCausalLM], tests/models/test_registry.py::test_registry_imports[LLaMAForCausalLM], tests/models/test_registry.py::test_registry_imports[LongcatFlashForCausalLM], tests/models/test_registry.py::test_registry_imports[MambaForCausalLM], tests/models/test_registry.py::test_registry_imports[FalconMambaForCausalLM], tests/models/test_registry.py::test_registry_imports[FalconH1ForCausalLM], tests/models/test_registry.py::test_registry_imports[Mamba2ForCausalLM], tests/models/test_registry.py::test_registry_imports[MiniCPMForCausalLM], tests/models/test_registry.py::test_registry_imports[MiniCPM3ForCausalLM], tests/models/test_registry.py::test_registry_imports[MistralForCausalLM], tests/models/test_registry.py::test_registry_imports[MixtralForCausalLM], tests/models/test_registry.py::test_registry_imports[MptForCausalLM], tests/models/test_registry.py::test_registry_imports[MPTForCausalLM], tests/models/test_registry.py::test_registry_imports[MiMoForCausalLM], tests/models/test_registry.py::test_registry_imports[NemotronForCausalLM], tests/models/test_registry.py::test_registry_imports[NemotronHForCausalLM], tests/models/test_registry.py::test_registry_imports[OlmoForCausalLM], tests/models/test_registry.py::test_registry_imports[Olmo2ForCausalLM], tests/models/test_registry.py::test_registry_imports[Olmo3ForCausalLM], tests/models/test_registry.py::test_registry_imports[OlmoeForCausalLM], tests/models/test_registry.py::test_registry_imports[OPTForCausalLM], tests/models/test_registry.py::test_registry_imports[OrionForCausalLM], tests/models/test_registry.py::test_registry_imports[PersimmonForCausalLM], tests/models/test_registry.py::test_registry_imports[PhiForCausalLM], tests/models/test_registry.py::test_registry_imports[Phi3ForCausalLM], tests/models/test_registry.py::test_registry_imports[PhiMoEForCausalLM], tests/models/test_registry.py::test_registry_imports[Plamo2ForCausalLM], tests/models/test_registry.py::test_registry_imports[QWenLMHeadModel], tests/models/test_registry.py::test_registry_imports[Qwen2ForCausalLM], tests/models/test_registry.py::test_registry_imports[Qwen2MoeForCausalLM], tests/models/test_registry.py::test_registry_imports[Qwen3ForCausalLM], tests/models/test_registry.py::test_registry_imports[Qwen3MoeForCausalLM], tests/models/test_registry.py::test_registry_imports[RWForCausalLM], tests/models/test_registry.py::test_registry_imports[SeedOssForCausalLM], tests/models/test_registry.py::test_registry_imports[Step3TextForCausalLM], tests/models/test_registry.py::test_registry_imports[StableLMEpochForCausalLM], tests/models/test_registry.py::test_registry_imports[StableLmForCausalLM], tests/models/test_registry.py::test_registry_imports[Starcoder2ForCausalLM], tests/models/test_registry.py::test_registry_imports[SolarForCausalLM], tests/models/test_registry.py::test_registry_imports[TeleChat2ForCausalLM], tests/models/test_registry.py::test_registry_imports[TeleFLMForCausalLM], tests/models/test_registry.py::test_registry_imports[XverseForCausalLM], tests/models/test_registry.py::test_registry_imports[Zamba2ForCausalLM], tests/models/test_registry.py::test_registry_imports[BertModel], tests/models/test_registry.py::test_registry_imports[Gemma2Model], tests/models/test_registry.py::test_registry_imports[Gemma3TextModel], tests/models/test_registry.py::test_registry_imports[GPT2ForSequenceClassification], tests/models/test_registry.py::test_registry_imports[GteModel], tests/models/test_registry.py::test_registry_imports[GteNewModel], tests/models/test_registry.py::test_registry_imports[InternLM2ForRewardModel], tests/models/test_registry.py::test_registry_imports[JambaForSequenceClassification], tests/models/test_registry.py::test_registry_imports[LlamaModel], tests/models/test_registry.py::test_registry_imports[MistralModel], tests/models/test_registry.py::test_registry_imports[ModernBertModel], tests/models/test_registry.py::test_registry_imports[NomicBertModel], tests/models/test_registry.py::test_registry_imports[Qwen2Model], tests/models/test_registry.py::test_registry_imports[Qwen2ForRewardModel], tests/models/test_registry.py::test_registry_imports[Qwen2ForProcessRewardModel], tests/models/test_registry.py::test_registry_imports[RobertaForMaskedLM], tests/models/test_registry.py::test_registry_imports[RobertaModel], tests/models/test_registry.py::test_registry_imports[XLMRobertaModel], tests/models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Phi3VForCausalLM], tests/models/test_registry.py::test_registry_imports[Qwen2VLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE], tests/models/test_registry.py::test_registry_imports[Terratorch], tests/models/test_registry.py::test_registry_imports[BertForSequenceClassification], tests/models/test_registry.py::test_registry_imports[BertForTokenClassification], tests/models/test_registry.py::test_registry_imports[GteNewForSequenceClassification], tests/models/test_registry.py::test_registry_imports[ModernBertForSequenceClassification], tests/models/test_registry.py::test_registry_imports[RobertaForSequenceClassification], tests/models/test_registry.py::test_registry_imports[XLMRobertaForSequenceClassification], tests/models/test_registry.py::test_registry_imports[JinaVLForRanking], tests/models/test_registry.py::test_registry_imports[AriaForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[AyaVisionForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Blip2ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[ChameleonForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Cohere2VisionForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[DeepseekVLV2ForCausalLM], tests/models/test_registry.py::test_registry_imports[DotsOCRForCausalLM], tests/models/test_registry.py::test_registry_imports[Ernie4_5_VLMoeForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[FuyuForCausalLM], tests/models/test_registry.py::test_registry_imports[Gemma3ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Gemma3nForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[GLM4VForCausalLM], tests/models/test_registry.py::test_registry_imports[Glm4vForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Glm4vMoeForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[GraniteSpeechForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[H2OVLChatModel], tests/models/test_registry.py::test_registry_imports[InternVLChatModel], tests/models/test_registry.py::test_registry_imports[NemotronH_Nano_VL_V2], tests/models/test_registry.py::test_registry_imports[InternS1ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[InternVLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Idefics3ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[SmolVLMForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[KeyeForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[KeyeVL1_5ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[RForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[KimiVLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Llama_Nemotron_Nano_VL], tests/models/test_registry.py::test_registry_imports[Llama4ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[LlavaForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[LlavaNextVideoForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[LlavaOnevisionForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[MantisForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[MiDashengLMModel], tests/models/test_registry.py::test_registry_imports[MiniMaxVL01ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[MiniCPMO], tests/models/test_registry.py::test_registry_imports[MiniCPMV], tests/models/test_registry.py::test_registry_imports[Mistral3ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[MolmoForCausalLM], tests/models/test_registry.py::test_registry_imports[NVLM_D], tests/models/test_registry.py::test_registry_imports[Ovis], tests/models/test_registry.py::test_registry_imports[Ovis2_5], tests/models/test_registry.py::test_registry_imports[PaliGemmaForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Phi4MMForCausalLM], tests/models/test_registry.py::test_registry_imports[Phi4MultimodalForCausalLM], tests/models/test_registry.py::test_registry_imports[PixtralForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[QwenVLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Qwen2_5_VLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Qwen2AudioForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Qwen2_5OmniModel], tests/models/test_registry.py::test_registry_imports[Qwen2_5OmniForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Qwen3VLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Qwen3VLMoeForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[SkyworkR1VChatModel], tests/models/test_registry.py::test_registry_imports[Step3VLForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[TarsierForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[Tarsier2ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[UltravoxModel], tests/models/test_registry.py::test_registry_imports[VoxtralForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[WhisperForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[MiMoMTPModel], tests/models/test_registry.py::test_registry_imports[EagleLlamaForCausalLM], tests/models/test_registry.py::test_registry_imports[EagleLlama4ForCausalLM], tests/models/test_registry.py::test_registry_imports[EagleMiniCPMForCausalLM], tests/models/test_registry.py::test_registry_imports[Eagle3LlamaForCausalLM], tests/models/test_registry.py::test_registry_imports[LlamaForCausalLMEagle3], tests/models/test_registry.py::test_registry_imports[Eagle3Qwen2_5vlForCausalLM], tests/models/test_registry.py::test_registry_imports[EagleDeepSeekMTPModel], tests/models/test_registry.py::test_registry_imports[DeepSeekMTPModel], tests/models/test_registry.py::test_registry_imports[ErnieMTPModel], tests/models/test_registry.py::test_registry_imports[LongCatFlashMTPModel], tests/models/test_registry.py::test_registry_imports[Glm4MoeMTPModel], tests/models/test_registry.py::test_registry_imports[MedusaModel], tests/models/test_registry.py::test_registry_imports[Qwen3NextMTP], tests/models/test_registry.py::test_registry_imports[SmolLM3ForCausalLM], tests/models/test_registry.py::test_registry_imports[Emu3ForConditionalGeneration], tests/models/test_registry.py::test_registry_imports[TransformersForCausalLM], tests/models/test_registry.py::test_registry_imports[TransformersForMultimodalLM], tests/models/test_registry.py::test_registry_imports[TransformersMoEForCausalLM], tests/models/test_registry.py::test_registry_imports[TransformersMoEForMultimodalLM], tests/models/test_registry.py::test_registry_imports[TransformersEmbeddingModel], tests/models/test_registry.py::test_registry_imports[TransformersForSequenceClassification], tests/models/test_registry.py::test_registry_imports[TransformersMoEForSequenceClassification], tests/models/test_registry.py::test_registry_imports[TransformersMoEEmbeddingModel], tests/models/test_registry.py::test_registry_model_property[LlamaForCausalLM-False-False-False], tests/models/test_registry.py::test_registry_model_property[LlavaForConditionalGeneration-True-True-False], tests/models/test_registry.py::test_registry_model_property[BertForSequenceClassification-False-False-True], tests/models/test_registry.py::test_registry_model_property[RobertaForSequenceClassification-False-False-True], tests/models/test_registry.py::test_registry_model_property[XLMRobertaForSequenceClassification-False-False-True], tests/models/test_registry.py::test_registry_is_pp[DeepseekV2ForCausalLM-True-False], tests/models/test_registry.py::test_registry_is_pp[Qwen2VLForConditionalGeneration-True-True], tests/models/test_registry.py::test_hf_registry_coverage 2025-10-10T01:34:44.2057325Z 2025-10-10T01:34:44.4720718Z models/test_registry.py::test_registry_imports[ApertusForCausalLM] PASSED 2025-10-10T01:34:44.7182500Z models/test_registry.py::test_registry_imports[AquilaModel] PASSED 2025-10-10T01:34:44.9545110Z models/test_registry.py::test_registry_imports[AquilaForCausalLM] PASSED 2025-10-10T01:34:45.1898898Z models/test_registry.py::test_registry_imports[ArceeForCausalLM] PASSED 2025-10-10T01:34:45.4416019Z models/test_registry.py::test_registry_imports[ArcticForCausalLM] PASSED 2025-10-10T01:34:45.6882765Z models/test_registry.py::test_registry_imports[MiniMaxForCausalLM] PASSED 2025-10-10T01:34:45.9257515Z models/test_registry.py::test_registry_imports[MiniMaxText01ForCausalLM] PASSED 2025-10-10T01:34:46.1611296Z models/test_registry.py::test_registry_imports[MiniMaxM1ForCausalLM] PASSED 2025-10-10T01:34:46.3968511Z models/test_registry.py::test_registry_imports[BaiChuanForCausalLM] PASSED 2025-10-10T01:34:46.6318035Z models/test_registry.py::test_registry_imports[BaichuanForCausalLM] PASSED 2025-10-10T01:34:46.8723598Z models/test_registry.py::test_registry_imports[BailingMoeForCausalLM] PASSED 2025-10-10T01:34:47.1093069Z models/test_registry.py::test_registry_imports[BailingMoeV2ForCausalLM] PASSED 2025-10-10T01:34:47.3709876Z models/test_registry.py::test_registry_imports[BambaForCausalLM] PASSED 2025-10-10T01:34:47.6182575Z models/test_registry.py::test_registry_imports[BloomForCausalLM] PASSED 2025-10-10T01:34:47.8605431Z models/test_registry.py::test_registry_imports[ChatGLMModel] SKIPPED 2025-10-10T01:34:48.0991514Z models/test_registry.py::test_registry_imports[ChatGLMForConditionalGeneration] PASSED 2025-10-10T01:34:48.3432830Z models/test_registry.py::test_registry_imports[CohereForCausalLM] PASSED 2025-10-10T01:34:48.5794735Z models/test_registry.py::test_registry_imports[Cohere2ForCausalLM] PASSED 2025-10-10T01:34:48.8186041Z models/test_registry.py::test_registry_imports[CwmForCausalLM] PASSED 2025-10-10T01:34:49.0677745Z models/test_registry.py::test_registry_imports[DbrxForCausalLM] PASSED 2025-10-10T01:34:49.3109580Z models/test_registry.py::test_registry_imports[DeciLMForCausalLM] PASSED 2025-10-10T01:34:49.5497560Z models/test_registry.py::test_registry_imports[DeepseekForCausalLM] PASSED 2025-10-10T01:34:49.7967253Z models/test_registry.py::test_registry_imports[DeepseekV2ForCausalLM] PASSED 2025-10-10T01:34:50.0368603Z models/test_registry.py::test_registry_imports[DeepseekV3ForCausalLM] PASSED 2025-10-10T01:34:50.2748035Z models/test_registry.py::test_registry_imports[DeepseekV32ForCausalLM] PASSED 2025-10-10T01:34:50.5161543Z models/test_registry.py::test_registry_imports[Dots1ForCausalLM] PASSED 2025-10-10T01:34:50.7548876Z models/test_registry.py::test_registry_imports[Ernie4_5ForCausalLM] PASSED 2025-10-10T01:34:51.0003587Z models/test_registry.py::test_registry_imports[Ernie4_5_MoeForCausalLM] PASSED 2025-10-10T01:34:51.2411981Z models/test_registry.py::test_registry_imports[ExaoneForCausalLM] PASSED 2025-10-10T01:34:51.4847895Z models/test_registry.py::test_registry_imports[Exaone4ForCausalLM] PASSED 2025-10-10T01:34:51.7258355Z models/test_registry.py::test_registry_imports[FalconForCausalLM] PASSED 2025-10-10T01:34:51.9679617Z models/test_registry.py::test_registry_imports[Fairseq2LlamaForCausalLM] PASSED 2025-10-10T01:34:52.2112297Z models/test_registry.py::test_registry_imports[GemmaForCausalLM] PASSED 2025-10-10T01:34:52.4519551Z models/test_registry.py::test_registry_imports[Gemma2ForCausalLM] PASSED 2025-10-10T01:34:52.6976147Z models/test_registry.py::test_registry_imports[Gemma3ForCausalLM] PASSED 2025-10-10T01:34:53.5509184Z models/test_registry.py::test_registry_imports[Gemma3nForCausalLM] PASSED 2025-10-10T01:34:53.8023556Z models/test_registry.py::test_registry_imports[Qwen3NextForCausalLM] SKIPPED 2025-10-10T01:34:54.0581405Z models/test_registry.py::test_registry_imports[GlmForCausalLM] PASSED 2025-10-10T01:34:54.3132723Z models/test_registry.py::test_registry_imports[Glm4ForCausalLM] PASSED 2025-10-10T01:34:54.5681643Z models/test_registry.py::test_registry_imports[Glm4MoeForCausalLM] PASSED 2025-10-10T01:34:54.8237824Z models/test_registry.py::test_registry_imports[GptOssForCausalLM] PASSED 2025-10-10T01:34:55.0845091Z models/test_registry.py::test_registry_imports[GPT2LMHeadModel] PASSED 2025-10-10T01:34:55.3403881Z models/test_registry.py::test_registry_imports[GPTBigCodeForCausalLM] PASSED 2025-10-10T01:34:55.5971976Z models/test_registry.py::test_registry_imports[GPTJForCausalLM] PASSED 2025-10-10T01:34:55.8525937Z models/test_registry.py::test_registry_imports[GPTNeoXForCausalLM] PASSED 2025-10-10T01:34:56.1094700Z models/test_registry.py::test_registry_imports[GraniteForCausalLM] PASSED 2025-10-10T01:34:56.3635324Z models/test_registry.py::test_registry_imports[GraniteMoeForCausalLM] PASSED 2025-10-10T01:34:56.6218473Z models/test_registry.py::test_registry_imports[GraniteMoeHybridForCausalLM] PASSED 2025-10-10T01:34:56.8736145Z models/test_registry.py::test_registry_imports[GraniteMoeSharedForCausalLM] PASSED 2025-10-10T01:34:57.1304864Z models/test_registry.py::test_registry_imports[GritLM] PASSED 2025-10-10T01:34:57.3859710Z models/test_registry.py::test_registry_imports[Grok1ModelForCausalLM] PASSED 2025-10-10T01:34:57.6386777Z models/test_registry.py::test_registry_imports[HunYuanMoEV1ForCausalLM] PASSED 2025-10-10T01:34:57.8907940Z models/test_registry.py::test_registry_imports[HunYuanDenseV1ForCausalLM] PASSED 2025-10-10T01:34:58.1503679Z models/test_registry.py::test_registry_imports[HCXVisionForCausalLM] PASSED 2025-10-10T01:34:58.4059241Z models/test_registry.py::test_registry_imports[InternLMForCausalLM] PASSED 2025-10-10T01:34:58.6595012Z models/test_registry.py::test_registry_imports[InternLM2ForCausalLM] PASSED 2025-10-10T01:34:58.9122911Z models/test_registry.py::test_registry_imports[InternLM2VEForCausalLM] PASSED 2025-10-10T01:34:59.1640113Z models/test_registry.py::test_registry_imports[InternLM3ForCausalLM] PASSED 2025-10-10T01:34:59.4191144Z models/test_registry.py::test_registry_imports[JAISLMHeadModel] PASSED 2025-10-10T01:34:59.6779925Z models/test_registry.py::test_registry_imports[JambaForCausalLM] PASSED 2025-10-10T01:34:59.9354617Z models/test_registry.py::test_registry_imports[Lfm2ForCausalLM] PASSED 2025-10-10T01:35:00.1874128Z models/test_registry.py::test_registry_imports[LlamaForCausalLM] PASSED 2025-10-10T01:35:00.4485747Z models/test_registry.py::test_registry_imports[Llama4ForCausalLM] PASSED 2025-10-10T01:35:00.7053202Z models/test_registry.py::test_registry_imports[LLaMAForCausalLM] PASSED 2025-10-10T01:35:00.9599829Z models/test_registry.py::test_registry_imports[LongcatFlashForCausalLM] PASSED 2025-10-10T01:35:01.2175058Z models/test_registry.py::test_registry_imports[MambaForCausalLM] PASSED 2025-10-10T01:35:01.4743998Z models/test_registry.py::test_registry_imports[FalconMambaForCausalLM] PASSED 2025-10-10T01:35:01.7325462Z models/test_registry.py::test_registry_imports[FalconH1ForCausalLM] PASSED 2025-10-10T01:35:01.9875548Z models/test_registry.py::test_registry_imports[Mamba2ForCausalLM] PASSED 2025-10-10T01:35:02.2423278Z models/test_registry.py::test_registry_imports[MiniCPMForCausalLM] PASSED 2025-10-10T01:35:02.4987804Z models/test_registry.py::test_registry_imports[MiniCPM3ForCausalLM] PASSED 2025-10-10T01:35:02.7531668Z models/test_registry.py::test_registry_imports[MistralForCausalLM] PASSED 2025-10-10T01:35:03.0093532Z models/test_registry.py::test_registry_imports[MixtralForCausalLM] PASSED 2025-10-10T01:35:03.2670310Z models/test_registry.py::test_registry_imports[MptForCausalLM] PASSED 2025-10-10T01:35:03.5486612Z models/test_registry.py::test_registry_imports[MPTForCausalLM] PASSED 2025-10-10T01:35:03.8097765Z models/test_registry.py::test_registry_imports[MiMoForCausalLM] PASSED 2025-10-10T01:35:04.0656267Z models/test_registry.py::test_registry_imports[NemotronForCausalLM] PASSED 2025-10-10T01:35:04.3207657Z models/test_registry.py::test_registry_imports[NemotronHForCausalLM] PASSED 2025-10-10T01:35:04.5817179Z models/test_registry.py::test_registry_imports[OlmoForCausalLM] PASSED 2025-10-10T01:35:04.8405437Z models/test_registry.py::test_registry_imports[Olmo2ForCausalLM] PASSED 2025-10-10T01:35:05.0962816Z models/test_registry.py::test_registry_imports[Olmo3ForCausalLM] PASSED 2025-10-10T01:35:05.3539269Z models/test_registry.py::test_registry_imports[OlmoeForCausalLM] PASSED 2025-10-10T01:35:05.6164304Z models/test_registry.py::test_registry_imports[OPTForCausalLM] PASSED 2025-10-10T01:35:05.8741067Z models/test_registry.py::test_registry_imports[OrionForCausalLM] PASSED 2025-10-10T01:35:06.1324097Z models/test_registry.py::test_registry_imports[PersimmonForCausalLM] PASSED 2025-10-10T01:35:06.3915564Z models/test_registry.py::test_registry_imports[PhiForCausalLM] PASSED 2025-10-10T01:35:06.6508898Z models/test_registry.py::test_registry_imports[Phi3ForCausalLM] PASSED 2025-10-10T01:35:06.9095049Z models/test_registry.py::test_registry_imports[PhiMoEForCausalLM] PASSED 2025-10-10T01:35:07.1654541Z models/test_registry.py::test_registry_imports[Plamo2ForCausalLM] SKIPPED 2025-10-10T01:35:07.4211399Z models/test_registry.py::test_registry_imports[QWenLMHeadModel] SKIPPED 2025-10-10T01:35:07.6815943Z models/test_registry.py::test_registry_imports[Qwen2ForCausalLM] PASSED 2025-10-10T01:35:07.9422491Z models/test_registry.py::test_registry_imports[Qwen2MoeForCausalLM] PASSED 2025-10-10T01:35:08.2014998Z models/test_registry.py::test_registry_imports[Qwen3ForCausalLM] PASSED 2025-10-10T01:35:08.4585588Z models/test_registry.py::test_registry_imports[Qwen3MoeForCausalLM] PASSED 2025-10-10T01:35:08.7187267Z models/test_registry.py::test_registry_imports[RWForCausalLM] PASSED 2025-10-10T01:35:08.9775549Z models/test_registry.py::test_registry_imports[SeedOssForCausalLM] PASSED 2025-10-10T01:35:09.2357931Z models/test_registry.py::test_registry_imports[Step3TextForCausalLM] PASSED 2025-10-10T01:35:09.4944114Z models/test_registry.py::test_registry_imports[StableLMEpochForCausalLM] PASSED 2025-10-10T01:35:09.7563883Z models/test_registry.py::test_registry_imports[StableLmForCausalLM] PASSED 2025-10-10T01:35:10.0195844Z models/test_registry.py::test_registry_imports[Starcoder2ForCausalLM] PASSED 2025-10-10T01:35:10.2780881Z models/test_registry.py::test_registry_imports[SolarForCausalLM] PASSED 2025-10-10T01:35:10.5346397Z models/test_registry.py::test_registry_imports[TeleChat2ForCausalLM] PASSED 2025-10-10T01:35:10.7969497Z models/test_registry.py::test_registry_imports[TeleFLMForCausalLM] PASSED 2025-10-10T01:35:11.0538706Z models/test_registry.py::test_registry_imports[XverseForCausalLM] PASSED 2025-10-10T01:35:11.3164611Z models/test_registry.py::test_registry_imports[Zamba2ForCausalLM] PASSED 2025-10-10T01:35:11.5792146Z models/test_registry.py::test_registry_imports[BertModel] PASSED 2025-10-10T01:35:11.8406435Z models/test_registry.py::test_registry_imports[Gemma2Model] PASSED 2025-10-10T01:35:12.0992758Z models/test_registry.py::test_registry_imports[Gemma3TextModel] PASSED 2025-10-10T01:35:12.0998895Z models/test_registry.py::test_registry_imports[GPT2ForSequenceClassification] WARNING 10-10 01:35:12 [interfaces_base.py:68] The model () is missing the `get_input_embeddings` method. 2025-10-10T01:35:12.1000690Z WARNING 10-10 01:35:12 [interfaces_base.py:68] The model () is missing the `get_input_embeddings` method. 2025-10-10T01:35:12.1002382Z WARNING 10-10 01:35:12 [interfaces_base.py:68] The model () is missing the `get_input_embeddings` method. 2025-10-10T01:35:12.3568293Z PASSED 2025-10-10T01:35:12.6146420Z models/test_registry.py::test_registry_imports[GteModel] PASSED 2025-10-10T01:35:12.8764433Z models/test_registry.py::test_registry_imports[GteNewModel] PASSED 2025-10-10T01:35:13.1338593Z models/test_registry.py::test_registry_imports[InternLM2ForRewardModel] PASSED 2025-10-10T01:35:13.3904328Z models/test_registry.py::test_registry_imports[JambaForSequenceClassification] PASSED 2025-10-10T01:35:13.6475726Z models/test_registry.py::test_registry_imports[LlamaModel] PASSED 2025-10-10T01:35:13.9110062Z models/test_registry.py::test_registry_imports[MistralModel] PASSED 2025-10-10T01:35:14.1729105Z models/test_registry.py::test_registry_imports[ModernBertModel] PASSED 2025-10-10T01:35:14.4303524Z models/test_registry.py::test_registry_imports[NomicBertModel] PASSED 2025-10-10T01:35:14.6883694Z models/test_registry.py::test_registry_imports[Qwen2Model] PASSED 2025-10-10T01:35:14.9508006Z models/test_registry.py::test_registry_imports[Qwen2ForRewardModel] SKIPPED 2025-10-10T01:35:15.2123643Z models/test_registry.py::test_registry_imports[Qwen2ForProcessRewardModel] SKIPPED 2025-10-10T01:35:15.4749583Z models/test_registry.py::test_registry_imports[RobertaForMaskedLM] PASSED 2025-10-10T01:35:15.7322191Z models/test_registry.py::test_registry_imports[RobertaModel] PASSED 2025-10-10T01:35:15.9931108Z models/test_registry.py::test_registry_imports[XLMRobertaModel] PASSED 2025-10-10T01:35:16.6833975Z models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration] PASSED 2025-10-10T01:35:16.9518444Z models/test_registry.py::test_registry_imports[Phi3VForCausalLM] SKIPPED 2025-10-10T01:35:17.2320656Z models/test_registry.py::test_registry_imports[Qwen2VLForConditionalGeneration] PASSED 2025-10-10T01:35:23.9478930Z models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE] PASSED 2025-10-10T01:35:24.3588911Z models/test_registry.py::test_registry_imports[Terratorch] PASSED 2025-10-10T01:35:24.7641969Z models/test_registry.py::test_registry_imports[BertForSequenceClassification] PASSED 2025-10-10T01:35:25.1701963Z models/test_registry.py::test_registry_imports[BertForTokenClassification] PASSED 2025-10-10T01:35:25.5780635Z models/test_registry.py::test_registry_imports[GteNewForSequenceClassification] PASSED 2025-10-10T01:35:25.9838121Z models/test_registry.py::test_registry_imports[ModernBertForSequenceClassification] PASSED 2025-10-10T01:35:26.4005581Z models/test_registry.py::test_registry_imports[RobertaForSequenceClassification] PASSED 2025-10-10T01:35:26.8081051Z models/test_registry.py::test_registry_imports[XLMRobertaForSequenceClassification] PASSED 2025-10-10T01:35:27.2139789Z models/test_registry.py::test_registry_imports[JinaVLForRanking] PASSED 2025-10-10T01:35:27.6466946Z models/test_registry.py::test_registry_imports[AriaForConditionalGeneration] PASSED 2025-10-10T01:35:28.0618297Z models/test_registry.py::test_registry_imports[AyaVisionForConditionalGeneration] PASSED 2025-10-10T01:35:28.4851171Z models/test_registry.py::test_registry_imports[Blip2ForConditionalGeneration] PASSED 2025-10-10T01:35:28.9007470Z models/test_registry.py::test_registry_imports[ChameleonForConditionalGeneration] PASSED 2025-10-10T01:35:29.3170652Z models/test_registry.py::test_registry_imports[Cohere2VisionForConditionalGeneration] PASSED 2025-10-10T01:35:29.7252218Z models/test_registry.py::test_registry_imports[DeepseekVLV2ForCausalLM] SKIPPED 2025-10-10T01:35:30.1381920Z models/test_registry.py::test_registry_imports[DotsOCRForCausalLM] PASSED 2025-10-10T01:35:30.5572601Z models/test_registry.py::test_registry_imports[Ernie4_5_VLMoeForConditionalGeneration] PASSED 2025-10-10T01:35:30.9818069Z models/test_registry.py::test_registry_imports[FuyuForCausalLM] PASSED 2025-10-10T01:35:31.3934924Z models/test_registry.py::test_registry_imports[Gemma3ForConditionalGeneration] PASSED 2025-10-10T01:35:31.8452403Z models/test_registry.py::test_registry_imports[Gemma3nForConditionalGeneration] PASSED 2025-10-10T01:35:32.2556039Z models/test_registry.py::test_registry_imports[GLM4VForCausalLM] PASSED 2025-10-10T01:35:32.6794720Z models/test_registry.py::test_registry_imports[Glm4vForConditionalGeneration] PASSED 2025-10-10T01:35:33.0907052Z models/test_registry.py::test_registry_imports[Glm4vMoeForConditionalGeneration] PASSED 2025-10-10T01:35:33.5003393Z models/test_registry.py::test_registry_imports[GraniteSpeechForConditionalGeneration] PASSED 2025-10-10T01:35:33.9119973Z models/test_registry.py::test_registry_imports[H2OVLChatModel] SKIPPED 2025-10-10T01:35:34.3243374Z models/test_registry.py::test_registry_imports[InternVLChatModel] PASSED 2025-10-10T01:35:34.7408752Z models/test_registry.py::test_registry_imports[NemotronH_Nano_VL_V2] PASSED 2025-10-10T01:35:35.1611393Z models/test_registry.py::test_registry_imports[InternS1ForConditionalGeneration] PASSED 2025-10-10T01:35:35.5707007Z models/test_registry.py::test_registry_imports[InternVLForConditionalGeneration] PASSED 2025-10-10T01:35:35.9949288Z models/test_registry.py::test_registry_imports[Idefics3ForConditionalGeneration] PASSED 2025-10-10T01:35:36.5005469Z models/test_registry.py::test_registry_imports[SmolVLMForConditionalGeneration] PASSED 2025-10-10T01:35:36.9157067Z models/test_registry.py::test_registry_imports[KeyeForConditionalGeneration] PASSED 2025-10-10T01:35:37.3288539Z models/test_registry.py::test_registry_imports[KeyeVL1_5ForConditionalGeneration] PASSED 2025-10-10T01:35:37.7647430Z models/test_registry.py::test_registry_imports[RForConditionalGeneration] PASSED 2025-10-10T01:35:38.1876839Z models/test_registry.py::test_registry_imports[KimiVLForConditionalGeneration] PASSED 2025-10-10T01:35:38.6042758Z models/test_registry.py::test_registry_imports[Llama_Nemotron_Nano_VL] PASSED 2025-10-10T01:35:39.0236759Z models/test_registry.py::test_registry_imports[Llama4ForConditionalGeneration] PASSED 2025-10-10T01:35:39.4368555Z models/test_registry.py::test_registry_imports[LlavaForConditionalGeneration] PASSED 2025-10-10T01:35:39.8587686Z models/test_registry.py::test_registry_imports[LlavaNextVideoForConditionalGeneration] PASSED 2025-10-10T01:35:40.2725341Z models/test_registry.py::test_registry_imports[LlavaOnevisionForConditionalGeneration] PASSED 2025-10-10T01:35:40.6837567Z models/test_registry.py::test_registry_imports[MantisForConditionalGeneration] SKIPPED 2025-10-10T01:35:41.2252447Z models/test_registry.py::test_registry_imports[MiDashengLMModel] PASSED 2025-10-10T01:35:41.6413639Z models/test_registry.py::test_registry_imports[MiniMaxVL01ForConditionalGeneration] PASSED 2025-10-10T01:35:42.0705277Z models/test_registry.py::test_registry_imports[MiniCPMO] PASSED 2025-10-10T01:35:42.4874522Z models/test_registry.py::test_registry_imports[MiniCPMV] PASSED 2025-10-10T01:35:42.9160292Z models/test_registry.py::test_registry_imports[Mistral3ForConditionalGeneration] PASSED 2025-10-10T01:35:43.3345373Z models/test_registry.py::test_registry_imports[MolmoForCausalLM] SKIPPED 2025-10-10T01:35:43.7497898Z models/test_registry.py::test_registry_imports[NVLM_D] PASSED 2025-10-10T01:35:44.1662400Z models/test_registry.py::test_registry_imports[Ovis] SKIPPED (`trans...) 2025-10-10T01:35:44.5930370Z models/test_registry.py::test_registry_imports[Ovis2_5] PASSED 2025-10-10T01:35:45.0146713Z models/test_registry.py::test_registry_imports[PaliGemmaForConditionalGeneration] PASSED 2025-10-10T01:35:45.4379533Z models/test_registry.py::test_registry_imports[Phi4MMForCausalLM] PASSED 2025-10-10T01:35:45.8844630Z models/test_registry.py::test_registry_imports[Phi4MultimodalForCausalLM] PASSED 2025-10-10T01:35:46.3021548Z models/test_registry.py::test_registry_imports[PixtralForConditionalGeneration] PASSED 2025-10-10T01:35:46.7208134Z models/test_registry.py::test_registry_imports[QwenVLForConditionalGeneration] PASSED 2025-10-10T01:35:47.1474119Z models/test_registry.py::test_registry_imports[Qwen2_5_VLForConditionalGeneration] PASSED 2025-10-10T01:35:47.5815016Z models/test_registry.py::test_registry_imports[Qwen2AudioForConditionalGeneration] PASSED 2025-10-10T01:35:48.0434935Z models/test_registry.py::test_registry_imports[Qwen2_5OmniModel] PASSED 2025-10-10T01:35:48.4632681Z models/test_registry.py::test_registry_imports[Qwen2_5OmniForConditionalGeneration] PASSED 2025-10-10T01:35:48.8801076Z models/test_registry.py::test_registry_imports[Qwen3VLForConditionalGeneration] SKIPPED 2025-10-10T01:35:49.3058290Z models/test_registry.py::test_registry_imports[Qwen3VLMoeForConditionalGeneration] SKIPPED 2025-10-10T01:35:49.7299051Z models/test_registry.py::test_registry_imports[SkyworkR1VChatModel] PASSED 2025-10-10T01:35:50.1557404Z models/test_registry.py::test_registry_imports[Step3VLForConditionalGeneration] PASSED 2025-10-10T01:35:50.5802881Z models/test_registry.py::test_registry_imports[TarsierForConditionalGeneration] PASSED 2025-10-10T01:35:51.0068036Z models/test_registry.py::test_registry_imports[Tarsier2ForConditionalGeneration] PASSED 2025-10-10T01:35:51.4310428Z models/test_registry.py::test_registry_imports[UltravoxModel] PASSED 2025-10-10T01:35:51.8965021Z models/test_registry.py::test_registry_imports[VoxtralForConditionalGeneration] PASSED 2025-10-10T01:35:52.3259134Z models/test_registry.py::test_registry_imports[WhisperForConditionalGeneration] PASSED 2025-10-10T01:35:52.7516634Z models/test_registry.py::test_registry_imports[MiMoMTPModel] PASSED 2025-10-10T01:35:53.1742004Z models/test_registry.py::test_registry_imports[EagleLlamaForCausalLM] PASSED 2025-10-10T01:35:53.5962431Z models/test_registry.py::test_registry_imports[EagleLlama4ForCausalLM] PASSED 2025-10-10T01:35:54.0173361Z models/test_registry.py::test_registry_imports[EagleMiniCPMForCausalLM] PASSED 2025-10-10T01:35:54.4525019Z models/test_registry.py::test_registry_imports[Eagle3LlamaForCausalLM] PASSED 2025-10-10T01:35:54.8787666Z models/test_registry.py::test_registry_imports[LlamaForCausalLMEagle3] PASSED 2025-10-10T01:35:55.2989745Z models/test_registry.py::test_registry_imports[Eagle3Qwen2_5vlForCausalLM] PASSED 2025-10-10T01:35:55.7205869Z models/test_registry.py::test_registry_imports[EagleDeepSeekMTPModel] PASSED 2025-10-10T01:35:56.1410169Z models/test_registry.py::test_registry_imports[DeepSeekMTPModel] PASSED 2025-10-10T01:35:56.5708052Z models/test_registry.py::test_registry_imports[ErnieMTPModel] PASSED 2025-10-10T01:35:56.9960431Z models/test_registry.py::test_registry_imports[LongCatFlashMTPModel] PASSED 2025-10-10T01:35:57.4235275Z models/test_registry.py::test_registry_imports[Glm4MoeMTPModel] PASSED 2025-10-10T01:35:57.8480963Z models/test_registry.py::test_registry_imports[MedusaModel] PASSED 2025-10-10T01:35:58.2690704Z models/test_registry.py::test_registry_imports[Qwen3NextMTP] SKIPPED 2025-10-10T01:35:58.6933171Z models/test_registry.py::test_registry_imports[SmolLM3ForCausalLM] PASSED 2025-10-10T01:35:59.1139097Z models/test_registry.py::test_registry_imports[Emu3ForConditionalGeneration] PASSED 2025-10-10T01:35:59.5429351Z models/test_registry.py::test_registry_imports[TransformersForCausalLM] PASSED 2025-10-10T01:35:59.9751986Z models/test_registry.py::test_registry_imports[TransformersForMultimodalLM] PASSED 2025-10-10T01:36:00.3990411Z models/test_registry.py::test_registry_imports[TransformersMoEForCausalLM] SKIPPED 2025-10-10T01:36:00.8281844Z models/test_registry.py::test_registry_imports[TransformersMoEForMultimodalLM] SKIPPED 2025-10-10T01:36:01.2555082Z models/test_registry.py::test_registry_imports[TransformersEmbeddingModel] SKIPPED 2025-10-10T01:36:01.6796274Z models/test_registry.py::test_registry_imports[TransformersForSequenceClassification] SKIPPED 2025-10-10T01:36:02.1031043Z models/test_registry.py::test_registry_imports[TransformersMoEForSequenceClassification] SKIPPED 2025-10-10T01:36:02.5250137Z models/test_registry.py::test_registry_imports[TransformersMoEEmbeddingModel] SKIPPED 2025-10-10T01:36:02.5442107Z models/test_registry.py::test_registry_model_property[LlamaForCausalLM-False-False-False] Fork a new process to run a test 970 2025-10-10T01:36:02.5453256Z Fork a new process to run a test 0 2025-10-10T01:36:03.0637274Z PASSED 2025-10-10T01:36:03.0823401Z models/test_registry.py::test_registry_model_property[LlavaForConditionalGeneration-True-True-False] Fork a new process to run a test 971 2025-10-10T01:36:03.0835445Z Fork a new process to run a test 0 2025-10-10T01:36:10.7105999Z PASSED 2025-10-10T01:36:10.7290984Z models/test_registry.py::test_registry_model_property[BertForSequenceClassification-False-False-True] Fork a new process to run a test 1022 2025-10-10T01:36:10.7303583Z Fork a new process to run a test 0 2025-10-10T01:36:17.8503290Z PASSED 2025-10-10T01:36:17.8688749Z models/test_registry.py::test_registry_model_property[RobertaForSequenceClassification-False-False-True] Fork a new process to run a test 1074 2025-10-10T01:36:17.8701929Z Fork a new process to run a test 0 2025-10-10T01:36:25.0118282Z PASSED 2025-10-10T01:36:25.0306808Z models/test_registry.py::test_registry_model_property[XLMRobertaForSequenceClassification-False-False-True] Fork a new process to run a test 1126 2025-10-10T01:36:25.0319565Z Fork a new process to run a test 0 2025-10-10T01:36:25.5499792Z PASSED 2025-10-10T01:36:25.5686388Z models/test_registry.py::test_registry_is_pp[DeepseekV2ForCausalLM-True-False] Fork a new process to run a test 1127 2025-10-10T01:36:25.5699785Z Fork a new process to run a test 0 2025-10-10T01:36:32.6321553Z PASSED 2025-10-10T01:36:32.6506798Z models/test_registry.py::test_registry_is_pp[Qwen2VLForConditionalGeneration-True-True] Fork a new process to run a test 1178 2025-10-10T01:36:32.6519688Z Fork a new process to run a test 0 2025-10-10T01:36:39.9678119Z PASSED 2025-10-10T01:36:40.3872939Z models/test_registry.py::test_hf_registry_coverage PASSED 2025-10-10T01:36:40.3873419Z 2025-10-10T01:36:40.3873698Z =============================== warnings summary =============================== 2025-10-10T01:36:40.3874416Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64 2025-10-10T01:36:40.3875863Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:36:40.3877209Z import pynvml # type: ignore[import] 2025-10-10T01:36:40.3877440Z 2025-10-10T01:36:40.3878081Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305 2025-10-10T01:36:40.3879882Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable. 2025-10-10T01:36:40.3880963Z ref_error: type[Exception] = jsonschema.RefResolutionError, 2025-10-10T01:36:40.3881182Z 2025-10-10T01:36:40.3881401Z tests/models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration] 2025-10-10T01:36:40.3881901Z tests/models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration] 2025-10-10T01:36:40.3882396Z tests/models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration] 2025-10-10T01:36:40.3883457Z tests/models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration] 2025-10-10T01:36:40.3884524Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/triton/runtime/autotuner.py:99: DeprecationWarning: warmup, rep, and use_cuda_graph parameters are deprecated. See https://github.com/triton-lang/triton/pull/4496 for details. 2025-10-10T01:36:40.3885404Z warnings.warn(("warmup, rep, and use_cuda_graph parameters are deprecated. See " 2025-10-10T01:36:40.3885675Z 2025-10-10T01:36:40.3885891Z tests/models/test_registry.py::test_registry_imports[LlavaNextForConditionalGeneration] 2025-10-10T01:36:40.3886621Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. 2025-10-10T01:36:40.3887374Z Overriding a previously registered kernel for the same operator and the same dispatch key 2025-10-10T01:36:40.3888514Z operator: xformers_python::_fused_allgather_and_linear_impl(Tensor scattered_input, Tensor[] weights, str process_group_name, Tensor(a3!)[] gathered_outputs, SymInt timeout_s, bool _wait, bool _memcpy, Tensor scale_scattered_input, Tensor?[] scales_weights) -> () 2025-10-10T01:36:40.3889515Z registered at /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/_library/custom_ops.py:922 2025-10-10T01:36:40.3889921Z dispatch key: ADInplaceOrView 2025-10-10T01:36:40.3890146Z previous kernel: no debug info 2025-10-10T01:36:40.3890900Z new kernel: registered at /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/_library/custom_ops.py:922 (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) 2025-10-10T01:36:40.3891629Z self.m.impl( 2025-10-10T01:36:40.3891732Z 2025-10-10T01:36:40.3891915Z tests/models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE] 2025-10-10T01:36:40.3892560Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pretrainedmodels/models/dpn.py:255: SyntaxWarning: "is" with 'str' literal. Did you mean "=="? 2025-10-10T01:36:40.3893096Z if block_type is 'proj': 2025-10-10T01:36:40.3893230Z 2025-10-10T01:36:40.3893413Z tests/models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE] 2025-10-10T01:36:40.3894051Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pretrainedmodels/models/dpn.py:258: SyntaxWarning: "is" with 'str' literal. Did you mean "=="? 2025-10-10T01:36:40.3894585Z elif block_type is 'down': 2025-10-10T01:36:40.3894721Z 2025-10-10T01:36:40.3894891Z tests/models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE] 2025-10-10T01:36:40.3895517Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pretrainedmodels/models/dpn.py:262: SyntaxWarning: "is" with 'str' literal. Did you mean "=="? 2025-10-10T01:36:40.3896055Z assert block_type is 'normal' 2025-10-10T01:36:40.3896407Z 2025-10-10T01:36:40.3896600Z tests/models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE] 2025-10-10T01:36:40.3897611Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/geopandas/_compat.py:7: DeprecationWarning: The 'shapely.geos' module is deprecated, and will be removed in a future version. All attributes of 'shapely.geos' are available directly from the top-level 'shapely' namespace (since shapely 2.0.0). 2025-10-10T01:36:40.3898521Z import shapely.geos 2025-10-10T01:36:40.3898645Z 2025-10-10T01:36:40.3898827Z tests/models/test_registry.py::test_registry_imports[PrithviGeoSpatialMAE] 2025-10-10T01:36:40.3900778Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/__init__.py:1615: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/Context.cpp:45.) 2025-10-10T01:36:40.3902782Z _C._set_float32_matmul_precision(precision) 2025-10-10T01:36:40.3902957Z 2025-10-10T01:36:40.3903198Z tests/models/test_registry.py::test_registry_model_property[LlamaForCausalLM-False-False-False] 2025-10-10T01:36:40.3903786Z tests/models/test_registry.py::test_registry_model_property[LlavaForConditionalGeneration-True-True-False] 2025-10-10T01:36:40.3904424Z tests/models/test_registry.py::test_registry_model_property[BertForSequenceClassification-False-False-True] 2025-10-10T01:36:40.3905076Z tests/models/test_registry.py::test_registry_model_property[RobertaForSequenceClassification-False-False-True] 2025-10-10T01:36:40.3905745Z tests/models/test_registry.py::test_registry_model_property[XLMRobertaForSequenceClassification-False-False-True] 2025-10-10T01:36:40.3906325Z tests/models/test_registry.py::test_registry_is_pp[DeepseekV2ForCausalLM-True-False] 2025-10-10T01:36:40.3906954Z tests/models/test_registry.py::test_registry_is_pp[Qwen2VLForConditionalGeneration-True-True] 2025-10-10T01:36:40.3907702Z /var/lib/jenkins/workspace/vllm/tests/utils.py:829: DeprecationWarning: This process (pid=913) is multi-threaded, use of fork() may lead to deadlocks in the child. 2025-10-10T01:36:40.3908270Z pid = os.fork() 2025-10-10T01:36:40.3908383Z 2025-10-10T01:36:40.3908552Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2025-10-10T01:36:40.3909198Z =========== 205 passed, 21 skipped, 19 warnings in 118.27s (0:01:58) =========== 2025-10-10T01:36:42.7223294Z 2025-10-10 01:36:42,721 [INFO] cli.lib.core.vllm.lib: Finish running step: pytest -v -s models/test_registry.py 2025-10-10T01:36:42.7224030Z 2025-10-10 01:36:42,721 [INFO] cli.lib.core.vllm.lib: Running step: pytest -v -s models/test_utils.py 2025-10-10T01:36:42.7224617Z 2025-10-10 01:36:42,721 [INFO] cli.lib.common.utils: [shell] pytest -v -s models/test_utils.py 2025-10-10T01:36:47.3941333Z INFO 10-10 01:36:47 [__init__.py:215] Automatically detected platform cuda. 2025-10-10T01:36:48.9408363Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset. 2025-10-10T01:36:48.9410358Z The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session" 2025-10-10T01:36:48.9411658Z 2025-10-10T01:36:48.9411902Z warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) 2025-10-10T01:36:48.9472115Z ============================= test session starts ============================== 2025-10-10T01:36:48.9472719Z platform linux -- Python 3.12.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/conda/envs/py_3.12/bin/python 2025-10-10T01:36:48.9565920Z cachedir: .pytest_cache 2025-10-10T01:36:48.9566840Z hypothesis profile 'ci' -> database=None, deadline=None, print_blob=True, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-10-10T01:36:48.9567476Z rootdir: /var/lib/jenkins/workspace/vllm 2025-10-10T01:36:48.9567767Z configfile: pyproject.toml 2025-10-10T01:36:48.9568778Z plugins: cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, xdist-3.3.1, xdoctest-1.3.0, asyncio-0.24.0, cov-6.3.0, buildkite-test-collector-0.1.9, schemathesis-3.39.15, anyio-4.6.2.post1, hydra-core-1.3.2, mock-3.14.0, subtests-0.14.1, timeout-2.3.1, hypothesis-6.131.0, shard-0.1.2, forked-1.6.0 2025-10-10T01:36:48.9569840Z asyncio: mode=Mode.STRICT, default_loop_scope=None 2025-10-10T01:36:50.8911466Z collecting ...  2025-10-10T01:36:50.8912191Z collecting 4 items  2025-10-10T01:36:50.8913412Z collected 4 items  2025-10-10T01:36:50.8914645Z Running 4 items in this shard: tests/models/test_utils.py::test_module_with_batchnorm_can_load, tests/models/test_utils.py::test_module_with_child_containing_batchnorm_can_autoload, tests/models/test_utils.py::test_module_skip_prefix, tests/models/test_utils.py::test_module_skip_substr 2025-10-10T01:36:50.8920473Z 2025-10-10T01:36:51.1251098Z models/test_utils.py::test_module_with_batchnorm_can_load PASSED 2025-10-10T01:36:51.3530524Z models/test_utils.py::test_module_with_child_containing_batchnorm_can_autoload PASSED 2025-10-10T01:36:51.5850782Z models/test_utils.py::test_module_skip_prefix PASSED 2025-10-10T01:36:51.8141558Z models/test_utils.py::test_module_skip_substr PASSED 2025-10-10T01:36:51.8142022Z 2025-10-10T01:36:51.8142288Z =============================== warnings summary =============================== 2025-10-10T01:36:51.8143371Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64 2025-10-10T01:36:51.8145194Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:36:51.8146654Z import pynvml # type: ignore[import] 2025-10-10T01:36:51.8146856Z 2025-10-10T01:36:51.8147158Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305 2025-10-10T01:36:51.8148520Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable. 2025-10-10T01:36:51.8149711Z ref_error: type[Exception] = jsonschema.RefResolutionError, 2025-10-10T01:36:51.8150053Z 2025-10-10T01:36:51.8150244Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2025-10-10T01:36:51.8150932Z ======================== 4 passed, 2 warnings in 2.87s ========================= 2025-10-10T01:36:53.0708735Z 2025-10-10 01:36:53,070 [INFO] cli.lib.core.vllm.lib: Finish running step: pytest -v -s models/test_utils.py 2025-10-10T01:36:53.0709318Z 2025-10-10 01:36:53,070 [INFO] cli.lib.core.vllm.lib: Running step: pytest -v -s models/test_vision.py 2025-10-10T01:36:53.0709809Z 2025-10-10 01:36:53,070 [INFO] cli.lib.common.utils: [shell] pytest -v -s models/test_vision.py 2025-10-10T01:36:57.7653675Z INFO 10-10 01:36:57 [__init__.py:215] Automatically detected platform cuda. 2025-10-10T01:36:59.3209233Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset. 2025-10-10T01:36:59.3211275Z The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session" 2025-10-10T01:36:59.3212616Z 2025-10-10T01:36:59.3212867Z warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) 2025-10-10T01:36:59.3273949Z ============================= test session starts ============================== 2025-10-10T01:36:59.3274554Z platform linux -- Python 3.12.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/conda/envs/py_3.12/bin/python 2025-10-10T01:36:59.3367790Z cachedir: .pytest_cache 2025-10-10T01:36:59.3368386Z hypothesis profile 'ci' -> database=None, deadline=None, print_blob=True, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-10-10T01:36:59.3369461Z rootdir: /var/lib/jenkins/workspace/vllm 2025-10-10T01:36:59.3369922Z configfile: pyproject.toml 2025-10-10T01:36:59.3370932Z plugins: cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, xdist-3.3.1, xdoctest-1.3.0, asyncio-0.24.0, cov-6.3.0, buildkite-test-collector-0.1.9, schemathesis-3.39.15, anyio-4.6.2.post1, hydra-core-1.3.2, mock-3.14.0, subtests-0.14.1, timeout-2.3.1, hypothesis-6.131.0, shard-0.1.2, forked-1.6.0 2025-10-10T01:36:59.3372010Z asyncio: mode=Mode.STRICT, default_loop_scope=None 2025-10-10T01:37:01.0642746Z collecting ... WARNING 10-10 01:37:01 [interface.py:488] Current platform cuda does not have '_pytestfixturefunction' attribute. 2025-10-10T01:37:01.3850344Z WARNING 10-10 01:37:01 [interface.py:488] Current platform cuda does not have '__test__' attribute. 2025-10-10T01:37:01.3851007Z WARNING 10-10 01:37:01 [interface.py:488] Current platform cuda does not have '__bases__' attribute. 2025-10-10T01:37:01.3852040Z WARNING 10-10 01:37:01 [interface.py:488] Current platform cuda does not have '__test__' attribute. 2025-10-10T01:37:01.3852680Z WARNING 10-10 01:37:01 [interface.py:488] Current platform cuda does not have '_schemathesis_test' attribute. 2025-10-10T01:37:01.3884356Z  2025-10-10T01:37:01.3889202Z collecting 19 items  2025-10-10T01:37:01.3889815Z collected 19 items  2025-10-10T01:37:01.3896995Z Running 19 items in this shard: tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers0-10-10-expected_features0], tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers1-10-10-expected_features1], tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers2-10-20-expected_features2], tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers3-10-20-expected_features3], tests/models/test_vision.py::test_run_dp_sharded_vision_model[1], tests/models/test_vision.py::test_run_dp_sharded_vision_model[4], tests/models/test_vision.py::test_run_dp_sharded_vision_model[5], tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes0-2-expected_shuffle_indices0-expected_gpu_sample_counts0-expected_grouped_sizes_per_gpu0-empty input], tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes1-4-expected_shuffle_indices1-expected_gpu_sample_counts1-expected_grouped_sizes_per_gpu1-fewer samples than GPUs], tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes2-1-expected_shuffle_indices2-expected_gpu_sample_counts2-expected_grouped_sizes_per_gpu2-single GPU], tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes3-2-expected_shuffle_indices3-expected_gpu_sample_counts3-expected_grouped_sizes_per_gpu3-balanced assignment], tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes4-2-expected_shuffle_indices4-expected_gpu_sample_counts4-expected_grouped_sizes_per_gpu4-unbalanced sizes], tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model[1], tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model[3], tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model[5], tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model_empty_input, tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model_uneven_load, tests/models/test_vision.py::test_simple_mrope_vision_model_spatial_merge[2], tests/models/test_vision.py::test_simple_mrope_vision_model_spatial_merge[4] 2025-10-10T01:37:01.3902988Z 2025-10-10T01:37:01.6298293Z models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers0-10-10-expected_features0] PASSED 2025-10-10T01:37:01.8650772Z models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers1-10-10-expected_features1] PASSED 2025-10-10T01:37:02.1019775Z models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers2-10-20-expected_features2] PASSED 2025-10-10T01:37:02.3371806Z models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers3-10-20-expected_features3] PASSED 2025-10-10T01:37:02.3380134Z models/test_vision.py::test_run_dp_sharded_vision_model[1] SKIPPED (...) 2025-10-10T01:37:02.3388169Z models/test_vision.py::test_run_dp_sharded_vision_model[4] SKIPPED (...) 2025-10-10T01:37:02.3395380Z models/test_vision.py::test_run_dp_sharded_vision_model[5] SKIPPED (...) 2025-10-10T01:37:02.5741615Z models/test_vision.py::test_get_load_balance_assignment_cases[sizes0-2-expected_shuffle_indices0-expected_gpu_sample_counts0-expected_grouped_sizes_per_gpu0-empty input] PASSED 2025-10-10T01:37:02.8080118Z models/test_vision.py::test_get_load_balance_assignment_cases[sizes1-4-expected_shuffle_indices1-expected_gpu_sample_counts1-expected_grouped_sizes_per_gpu1-fewer samples than GPUs] PASSED 2025-10-10T01:37:03.0469477Z models/test_vision.py::test_get_load_balance_assignment_cases[sizes2-1-expected_shuffle_indices2-expected_gpu_sample_counts2-expected_grouped_sizes_per_gpu2-single GPU] PASSED 2025-10-10T01:37:03.2822579Z models/test_vision.py::test_get_load_balance_assignment_cases[sizes3-2-expected_shuffle_indices3-expected_gpu_sample_counts3-expected_grouped_sizes_per_gpu3-balanced assignment] PASSED 2025-10-10T01:37:03.5445643Z models/test_vision.py::test_get_load_balance_assignment_cases[sizes4-2-expected_shuffle_indices4-expected_gpu_sample_counts4-expected_grouped_sizes_per_gpu4-unbalanced sizes] PASSED 2025-10-10T01:37:03.5451219Z models/test_vision.py::test_run_dp_sharded_mrope_vision_model[1] SKIPPED 2025-10-10T01:37:03.5458283Z models/test_vision.py::test_run_dp_sharded_mrope_vision_model[3] SKIPPED 2025-10-10T01:37:03.5464729Z models/test_vision.py::test_run_dp_sharded_mrope_vision_model[5] SKIPPED 2025-10-10T01:37:03.5471310Z models/test_vision.py::test_run_dp_sharded_mrope_vision_model_empty_input SKIPPED 2025-10-10T01:37:03.5477661Z models/test_vision.py::test_run_dp_sharded_mrope_vision_model_uneven_load SKIPPED 2025-10-10T01:37:03.9946668Z models/test_vision.py::test_simple_mrope_vision_model_spatial_merge[2] PASSED 2025-10-10T01:37:04.2355514Z models/test_vision.py::test_simple_mrope_vision_model_spatial_merge[4] PASSED 2025-10-10T01:37:04.2355955Z 2025-10-10T01:37:04.2356255Z =============================== warnings summary =============================== 2025-10-10T01:37:04.2356792Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64 2025-10-10T01:37:04.2358215Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:37:04.2359444Z import pynvml # type: ignore[import] 2025-10-10T01:37:04.2359607Z 2025-10-10T01:37:04.2359888Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305 2025-10-10T01:37:04.2361041Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable. 2025-10-10T01:37:04.2362089Z ref_error: type[Exception] = jsonschema.RefResolutionError, 2025-10-10T01:37:04.2362309Z 2025-10-10T01:37:04.2362491Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2025-10-10T01:37:04.2363083Z ================== 11 passed, 8 skipped, 2 warnings in 4.91s =================== 2025-10-10T01:37:05.6419309Z 2025-10-10 01:37:05,641 [INFO] cli.lib.core.vllm.lib: Finish running step: pytest -v -s models/test_vision.py 2025-10-10T01:37:05.6420484Z 2025-10-10 01:37:05,641 [INFO] cli.lib.core.vllm.lib: Running step: pytest -v -s models/test_initialization.py 2025-10-10T01:37:05.6421043Z 2025-10-10 01:37:05,641 [INFO] cli.lib.common.utils: [shell] pytest -v -s models/test_initialization.py 2025-10-10T01:37:10.3273408Z INFO 10-10 01:37:10 [__init__.py:215] Automatically detected platform cuda. 2025-10-10T01:37:11.8654192Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset. 2025-10-10T01:37:11.8656182Z The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session" 2025-10-10T01:37:11.8657524Z 2025-10-10T01:37:11.8658123Z warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) 2025-10-10T01:37:11.8718234Z ============================= test session starts ============================== 2025-10-10T01:37:11.8718815Z platform linux -- Python 3.12.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/conda/envs/py_3.12/bin/python 2025-10-10T01:37:11.8812010Z cachedir: .pytest_cache 2025-10-10T01:37:11.8812739Z hypothesis profile 'ci' -> database=None, deadline=None, print_blob=True, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-10-10T01:37:11.8813363Z rootdir: /var/lib/jenkins/workspace/vllm 2025-10-10T01:37:11.8813663Z configfile: pyproject.toml 2025-10-10T01:37:11.8814687Z plugins: cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, xdist-3.3.1, xdoctest-1.3.0, asyncio-0.24.0, cov-6.3.0, buildkite-test-collector-0.1.9, schemathesis-3.39.15, anyio-4.6.2.post1, hydra-core-1.3.2, mock-3.14.0, subtests-0.14.1, timeout-2.3.1, hypothesis-6.131.0, shard-0.1.2, forked-1.6.0 2025-10-10T01:37:11.8815571Z asyncio: mode=Mode.STRICT, default_loop_scope=None 2025-10-10T01:37:13.9218906Z collecting ...  2025-10-10T01:37:13.9241881Z collecting 222 items  2025-10-10T01:37:13.9242833Z collected 222 items  2025-10-10T01:37:13.9305212Z Running 222 items in this shard: tests/models/test_initialization.py::test_can_initialize_small_subset[LlavaForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_small_subset[Llama4ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_small_subset[BertForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_small_subset[Gemma3nForCausalLM], tests/models/test_initialization.py::test_can_initialize_small_subset[JinaVLForRanking], tests/models/test_initialization.py::test_can_initialize_small_subset[InternVLChatModel], tests/models/test_initialization.py::test_can_initialize_small_subset[InternLM2ForRewardModel], tests/models/test_initialization.py::test_can_initialize_small_subset[TransformersForMultimodalLM], tests/models/test_initialization.py::test_can_initialize_small_subset[PrithviGeoSpatialMAE], tests/models/test_initialization.py::test_can_initialize_small_subset[UltravoxModel], tests/models/test_initialization.py::test_can_initialize_small_subset[DeepSeekMTPModel], tests/models/test_initialization.py::test_can_initialize_small_subset[XLMRobertaModel], tests/models/test_initialization.py::test_can_initialize_large_subset[Olmo2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ErnieMTPModel], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Glm4MoeMTPModel], tests/models/test_initialization.py::test_can_initialize_large_subset[GptOssForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[SmolVLMForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[LlavaNextForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Step3TextForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[BaichuanForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[TarsierForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Eagle3LlamaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[CohereForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniMaxM1ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[OlmoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersEmbeddingModel], tests/models/test_initialization.py::test_can_initialize_large_subset[HCXVisionForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MistralForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ArcticForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[StableLmForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Emu3ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Exaone4ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Glm4ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[BailingMoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Gemma3TextModel], tests/models/test_initialization.py::test_can_initialize_large_subset[Plamo2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[WhisperForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniCPMForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[NemotronHForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5_MoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MambaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[LlavaNextVideoForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[GemmaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniMaxVL01ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Gemma2Model], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5OmniModel], tests/models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeHybridForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Llama_Nemotron_Nano_VL], tests/models/test_initialization.py::test_can_initialize_large_subset[XverseForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[InternS1ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[PaliGemmaForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[RForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[DeciLMForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen3NextForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Gemma3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Gemma3nForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[QwenVLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[LlamaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Ovis], tests/models/test_initialization.py::test_can_initialize_large_subset[MedusaModel], tests/models/test_initialization.py::test_can_initialize_large_subset[Glm4vForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[InternLMForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[NomicBertModel], tests/models/test_initialization.py::test_can_initialize_large_subset[FalconMambaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[AyaVisionForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen3NextMTP], tests/models/test_initialization.py::test_can_initialize_large_subset[FalconH1ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MPTForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[NVLM_D], tests/models/test_initialization.py::test_can_initialize_large_subset[GPT2LMHeadModel], tests/models/test_initialization.py::test_can_initialize_large_subset[AriaForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[OPTForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[BambaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Cohere2VisionForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[HunYuanMoEV1ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Fairseq2LlamaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[SkyworkR1VChatModel], tests/models/test_initialization.py::test_can_initialize_large_subset[InternLM2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5OmniForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[ChatGLMForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[EagleDeepSeekMTPModel], tests/models/test_initialization.py::test_can_initialize_large_subset[PersimmonForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Eagle3Qwen2_5vlForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2AudioForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[ExaoneForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[RWForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ApertusForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[OlmoForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[PixtralForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[LLaMAForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[JambaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[BailingMoeV2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[CwmForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[TeleFLMForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[GteNewModel], tests/models/test_initialization.py::test_can_initialize_large_subset[DotsOCRForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[RobertaForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[BloomForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[KeyeForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5_VLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Idefics3ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Lfm2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Dots1ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[JAISLMHeadModel], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniCPM3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEEmbeddingModel], tests/models/test_initialization.py::test_can_initialize_large_subset[MistralModel], tests/models/test_initialization.py::test_can_initialize_large_subset[DeepseekForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[GPTNeoXForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2VLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[KeyeVL1_5ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[HunYuanDenseV1ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniMaxForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[EagleMiniCPMForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Olmo3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[NemotronH_Nano_VL_V2], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2MoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[GraniteSpeechForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[AquilaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Glm4MoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[PhiMoEForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForRewardModel], tests/models/test_initialization.py::test_can_initialize_large_subset[GPT2ForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[ArceeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Zamba2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Step3VLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Llama4ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[GPTBigCodeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[GlmForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Starcoder2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ChatGLMModel], tests/models/test_initialization.py::test_can_initialize_large_subset[GraniteForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Phi3VForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Tarsier2ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[XLMRobertaForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[GteNewForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[BaiChuanForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Mistral3ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEForMultimodalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[LongCatFlashMTPModel], tests/models/test_initialization.py::test_can_initialize_large_subset[InternVLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[MptForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Phi3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[RobertaModel], tests/models/test_initialization.py::test_can_initialize_large_subset[DeepseekV2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[SeedOssForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[OrionForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ModernBertModel], tests/models/test_initialization.py::test_can_initialize_large_subset[MiMoForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[TeleChat2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MiDashengLMModel], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2Model], tests/models/test_initialization.py::test_can_initialize_large_subset[BertForTokenClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[InternLM3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[EagleLlamaForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[LlamaForCausalLMEagle3], tests/models/test_initialization.py::test_can_initialize_large_subset[MiMoMTPModel], tests/models/test_initialization.py::test_can_initialize_large_subset[GPTJForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ModernBertForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[GLM4VForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[DeepseekVLV2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[ChameleonForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[PhiForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Blip2ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[MixtralForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5_VLMoeForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[TransformersForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[Terratorch], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniMaxText01ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Cohere2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[EagleLlama4ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[NemotronForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[DbrxForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[JambaForSequenceClassification], tests/models/test_initialization.py::test_can_initialize_large_subset[MantisForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[KimiVLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Ovis2_5], tests/models/test_initialization.py::test_can_initialize_large_subset[Phi4MMForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[H2OVLChatModel], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen3MoeForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[RobertaForMaskedLM], tests/models/test_initialization.py::test_can_initialize_large_subset[LongcatFlashForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Gemma2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen3VLForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[LlamaModel], tests/models/test_initialization.py::test_can_initialize_large_subset[Phi4MultimodalForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Glm4vMoeForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[BertModel], tests/models/test_initialization.py::test_can_initialize_large_subset[GritLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForProcessRewardModel], tests/models/test_initialization.py::test_can_initialize_large_subset[InternLM2VEForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MolmoForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Grok1ModelForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[SmolLM3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[Mamba2ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[FalconForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[DeepseekV32ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[SolarForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniCPMV], tests/models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[MiniCPMO], tests/models/test_initialization.py::test_can_initialize_large_subset[DeepseekV3ForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[GteModel], tests/models/test_initialization.py::test_can_initialize_large_subset[LlavaOnevisionForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Qwen3VLMoeForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[QWenLMHeadModel], tests/models/test_initialization.py::test_can_initialize_large_subset[FuyuForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[AquilaModel], tests/models/test_initialization.py::test_can_initialize_large_subset[StableLMEpochForCausalLM], tests/models/test_initialization.py::test_can_initialize_large_subset[VoxtralForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[Gemma3ForConditionalGeneration], tests/models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeSharedForCausalLM], tests/models/test_initialization.py::test_implicit_converted_models[GemmaForSequenceClassification], tests/models/test_initialization.py::test_implicit_converted_models[LlamaForSequenceClassification], tests/models/test_initialization.py::test_implicit_converted_models[Qwen2ForSequenceClassification], tests/models/test_initialization.py::test_implicit_converted_models[Qwen3ForSequenceClassification] 2025-10-10T01:37:13.9365139Z 2025-10-10T01:37:13.9387804Z models/test_initialization.py::test_can_initialize_small_subset[LlavaForConditionalGeneration] Fork a new process to run a test 1396 2025-10-10T01:37:13.9397860Z Fork a new process to run a test 0 2025-10-10T01:37:13.9669231Z INFO 10-10 01:37:13 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlavaForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'llava-hf/llava-1.5-7b-hf'} 2025-10-10T01:37:14.0521943Z 2025-10-10T01:37:14.0523557Z config.json: 0% 0.00/950 [00:00", line 1387, in _gcd_import 2025-10-10T01:37:15.1885501Z 2025-10-10T01:37:15.1885759Z ERROR 10-10 01:37:15 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:37:15.1886099Z 2025-10-10T01:37:15.1886402Z ERROR 10-10 01:37:15 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:15.1887061Z 2025-10-10T01:37:15.1887318Z ERROR 10-10 01:37:15 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:37:15.1887662Z 2025-10-10T01:37:15.1887926Z ERROR 10-10 01:37:15 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:37:15.1888221Z 2025-10-10T01:37:15.1888447Z ERROR 10-10 01:37:15 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:15.1888761Z 2025-10-10T01:37:15.1889121Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava.py", line 40, in 2025-10-10T01:37:15.1889564Z 2025-10-10T01:37:15.1889778Z ERROR 10-10 01:37:15 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T01:37:15.1890061Z 2025-10-10T01:37:15.1890420Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:37:15.1890854Z 2025-10-10T01:37:15.1891153Z ERROR 10-10 01:37:15 [registry.py:542] from xformers import ops as xops 2025-10-10T01:37:15.1891374Z 2025-10-10T01:37:15.1891704Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:37:15.1892089Z 2025-10-10T01:37:15.1892206Z ERROR 10-10 01:37:15 [registry.py:542] from .fmha import ( 2025-10-10T01:37:15.1892403Z 2025-10-10T01:37:15.1892742Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:37:15.1893155Z 2025-10-10T01:37:15.1893386Z ERROR 10-10 01:37:15 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:37:15.1893691Z 2025-10-10T01:37:15.1894045Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:37:15.1894479Z 2025-10-10T01:37:15.1894644Z ERROR 10-10 01:37:15 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:37:15.1894876Z 2025-10-10T01:37:15.1895017Z ERROR 10-10 01:37:15 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:15.1895226Z 2025-10-10T01:37:15.1895569Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:37:15.1895961Z 2025-10-10T01:37:15.1896294Z ERROR 10-10 01:37:15 [registry.py:542] value = func() 2025-10-10T01:37:15.1896507Z 2025-10-10T01:37:15.1896617Z ERROR 10-10 01:37:15 [registry.py:542] ^^^^^^ 2025-10-10T01:37:15.1896793Z 2025-10-10T01:37:15.1897134Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:37:15.1897549Z 2025-10-10T01:37:15.1897746Z ERROR 10-10 01:37:15 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:37:15.1898010Z 2025-10-10T01:37:15.1898157Z ERROR 10-10 01:37:15 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:15.1898381Z 2025-10-10T01:37:15.1898748Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:15.1899176Z 2025-10-10T01:37:15.1899326Z ERROR 10-10 01:37:15 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:37:15.1899581Z 2025-10-10T01:37:15.1899711Z ERROR 10-10 01:37:15 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:15.1899931Z 2025-10-10T01:37:15.1900288Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:15.1901060Z 2025-10-10T01:37:15.1901340Z ERROR 10-10 01:37:15 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:37:15.1901689Z 2025-10-10T01:37:15.1901834Z ERROR 10-10 01:37:15 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:37:15.1902077Z 2025-10-10T01:37:15.1902478Z ERROR 10-10 01:37:15 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:15.1902878Z 2025-10-10T01:37:15.1903002Z ERROR 10-10 01:37:15 [registry.py:542] raise RuntimeError( 2025-10-10T01:37:15.1903189Z 2025-10-10T01:37:15.1903586Z ERROR 10-10 01:37:15 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:15.5702554Z FAILED 2025-10-10T01:37:15.5828522Z models/test_initialization.py::test_can_initialize_small_subset[Llama4ForConditionalGeneration] Fork a new process to run a test 1422 2025-10-10T01:37:15.5837980Z Fork a new process to run a test 0 2025-10-10T01:37:15.6115161Z INFO 10-10 01:37:15 [utils.py:233] non-default args: {'load_format': 'dummy', 'max_model_len': 10240, 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Llama4ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'meta-llama/Llama-4-Scout-17B-16E-Instruct'} 2025-10-10T01:37:15.7046342Z 2025-10-10T01:37:15.7046848Z config.json: 0% 0.00/2.18k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:37:25.3869081Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] EngineCore failed to start. 2025-10-10T01:37:25.3869526Z 2025-10-10T01:37:25.3869933Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] Traceback (most recent call last): 2025-10-10T01:37:25.3870292Z 2025-10-10T01:37:25.3870985Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:25.3871586Z 2025-10-10T01:37:25.3871982Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:25.3872392Z 2025-10-10T01:37:25.3872771Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3873128Z 2025-10-10T01:37:25.3873746Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:25.3874323Z 2025-10-10T01:37:25.3874753Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:25.3875169Z 2025-10-10T01:37:25.3875782Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:25.3876334Z 2025-10-10T01:37:25.3876742Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:25.3877463Z 2025-10-10T01:37:25.3877781Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3878070Z 2025-10-10T01:37:25.3878613Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:25.3879210Z 2025-10-10T01:37:25.3879481Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] self._init_executor() 2025-10-10T01:37:25.3879753Z 2025-10-10T01:37:25.3880294Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:25.3880809Z 2025-10-10T01:37:25.3881133Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:25.3881461Z 2025-10-10T01:37:25.3882450Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:25.3883132Z 2025-10-10T01:37:25.3883672Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:25.3884068Z 2025-10-10T01:37:25.3884498Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3884894Z 2025-10-10T01:37:25.3885677Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:25.3886231Z 2025-10-10T01:37:25.3886549Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:37:25.3886824Z 2025-10-10T01:37:25.3887103Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3887404Z 2025-10-10T01:37:25.3887949Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:25.3888438Z 2025-10-10T01:37:25.3888737Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:25.3889033Z 2025-10-10T01:37:25.3889304Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3889586Z 2025-10-10T01:37:25.3890115Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:25.3890606Z 2025-10-10T01:37:25.3890924Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:37:25.3891227Z 2025-10-10T01:37:25.3891515Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3891788Z 2025-10-10T01:37:25.3892237Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:25.3892654Z 2025-10-10T01:37:25.3892999Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:25.3893331Z 2025-10-10T01:37:25.3893621Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3894013Z 2025-10-10T01:37:25.3894374Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:37:25.3894711Z 2025-10-10T01:37:25.3895058Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:37:25.3895408Z 2025-10-10T01:37:25.3895785Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:25.3896533Z 2025-10-10T01:37:25.3896934Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:37:25.3897296Z 2025-10-10T01:37:25.3897684Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:37:25.3898044Z 2025-10-10T01:37:25.3898581Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:25.3898967Z 2025-10-10T01:37:25.3899486Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:25.3899966Z 2025-10-10T01:37:25.3900330Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:25.3900677Z 2025-10-10T01:37:25.3901208Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:25.3901689Z 2025-10-10T01:37:25.3902082Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:25.3902442Z 2025-10-10T01:37:25.3903004Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:25.3903514Z 2025-10-10T01:37:25.3903822Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:37:25.3904123Z 2025-10-10T01:37:25.3904766Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:25.3905343Z 2025-10-10T01:37:25.3905749Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:25.3906101Z 2025-10-10T01:37:25.3906376Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3906636Z 2025-10-10T01:37:25.3907208Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:25.3907738Z 2025-10-10T01:37:25.3908048Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:37:25.3908342Z 2025-10-10T01:37:25.3908634Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3908921Z 2025-10-10T01:37:25.3909546Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:25.3910249Z 2025-10-10T01:37:25.3910559Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:37:25.3910887Z 2025-10-10T01:37:25.3911158Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3911430Z 2025-10-10T01:37:25.3912032Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:25.3912562Z 2025-10-10T01:37:25.3912905Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:25.3913217Z 2025-10-10T01:37:25.3913507Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3913780Z 2025-10-10T01:37:25.3914401Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:25.3914888Z 2025-10-10T01:37:25.3915189Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:37:25.3915480Z 2025-10-10T01:37:25.3915745Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3916023Z 2025-10-10T01:37:25.3916548Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:25.3917035Z 2025-10-10T01:37:25.3917360Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:37:25.3917671Z 2025-10-10T01:37:25.3917910Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:37:25.3918159Z 2025-10-10T01:37:25.3918666Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:25.3919229Z 2025-10-10T01:37:25.3919498Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] raise RuntimeError( 2025-10-10T01:37:25.3919757Z 2025-10-10T01:37:25.3920337Z (EngineCore_DP0 pid=1501) ERROR 10-10 01:37:25 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:25.3921034Z (EngineCore_DP0 pid=1501) Process EngineCore_DP0: 2025-10-10T01:37:25.3921443Z (EngineCore_DP0 pid=1501) Traceback (most recent call last): 2025-10-10T01:37:25.3922063Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:37:25.3922591Z (EngineCore_DP0 pid=1501) self.run() 2025-10-10T01:37:25.3923159Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:37:25.3923739Z (EngineCore_DP0 pid=1501) self._target(*self._args, **self._kwargs) 2025-10-10T01:37:25.3924428Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:37:25.3924987Z (EngineCore_DP0 pid=1501) raise e 2025-10-10T01:37:25.3925588Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:25.3926360Z (EngineCore_DP0 pid=1501) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:25.3926822Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3927458Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:25.3928119Z (EngineCore_DP0 pid=1501) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:25.3928789Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:25.3929413Z (EngineCore_DP0 pid=1501) self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:25.3929867Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3930525Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:25.3931194Z (EngineCore_DP0 pid=1501) self._init_executor() 2025-10-10T01:37:25.3931872Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:25.3932571Z (EngineCore_DP0 pid=1501) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:25.3933306Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:25.3934030Z (EngineCore_DP0 pid=1501) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:25.3934546Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3935208Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:25.3935809Z (EngineCore_DP0 pid=1501) return func(*args, **kwargs) 2025-10-10T01:37:25.3936203Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3936844Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:25.3937486Z (EngineCore_DP0 pid=1501) worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:25.3937914Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3938592Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:25.3939265Z (EngineCore_DP0 pid=1501) module = importlib.import_module(module_name) 2025-10-10T01:37:25.3939718Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3940315Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:25.3940952Z (EngineCore_DP0 pid=1501) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:25.3941462Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3941969Z (EngineCore_DP0 pid=1501) File "", line 1387, in _gcd_import 2025-10-10T01:37:25.3942532Z (EngineCore_DP0 pid=1501) File "", line 1360, in _find_and_load 2025-10-10T01:37:25.3943135Z (EngineCore_DP0 pid=1501) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:25.3943823Z (EngineCore_DP0 pid=1501) File "", line 935, in _load_unlocked 2025-10-10T01:37:25.3944418Z (EngineCore_DP0 pid=1501) File "", line 999, in exec_module 2025-10-10T01:37:25.3945039Z (EngineCore_DP0 pid=1501) File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:25.3945779Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:25.3946474Z (EngineCore_DP0 pid=1501) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:25.3947209Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:25.3947944Z (EngineCore_DP0 pid=1501) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:25.3948832Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:25.3949539Z (EngineCore_DP0 pid=1501) class FlashAttentionMetadataBuilder( 2025-10-10T01:37:25.3950338Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:25.3951173Z (EngineCore_DP0 pid=1501) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:25.3951659Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3952358Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:25.3953054Z (EngineCore_DP0 pid=1501) if not is_fa_version_supported(fa_version): 2025-10-10T01:37:25.3953505Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3954260Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:25.3954967Z (EngineCore_DP0 pid=1501) return _is_fa2_supported(device)[0] 2025-10-10T01:37:25.3955387Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3956115Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:25.3956844Z (EngineCore_DP0 pid=1501) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:25.3957306Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3957990Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:25.3958639Z (EngineCore_DP0 pid=1501) prop = get_device_properties(device) 2025-10-10T01:37:25.3959143Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:25.3959820Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:25.3960495Z (EngineCore_DP0 pid=1501) _lazy_init() # will define _get_device_properties 2025-10-10T01:37:25.3960900Z (EngineCore_DP0 pid=1501) ^^^^^^^^^^^^ 2025-10-10T01:37:25.3961502Z (EngineCore_DP0 pid=1501) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:25.3962175Z (EngineCore_DP0 pid=1501) raise RuntimeError( 2025-10-10T01:37:25.3962877Z (EngineCore_DP0 pid=1501) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:25.8204102Z FAILED 2025-10-10T01:37:25.8330917Z models/test_initialization.py::test_can_initialize_small_subset[BertForSequenceClassification] Fork a new process to run a test 1505 2025-10-10T01:37:25.8342099Z Fork a new process to run a test 0 2025-10-10T01:37:25.8611755Z INFO 10-10 01:37:25 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BertForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'cross-encoder/ms-marco-MiniLM-L-6-v2'} 2025-10-10T01:37:26.1020148Z 2025-10-10T01:37:26.1020953Z config.json: 0% 0.00/794 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:37:27.5462745Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] EngineCore failed to start. 2025-10-10T01:37:27.5463383Z 2025-10-10T01:37:27.5464036Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] Traceback (most recent call last): 2025-10-10T01:37:27.5464657Z 2025-10-10T01:37:27.5465672Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:27.5466314Z 2025-10-10T01:37:27.5466727Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:27.5467120Z 2025-10-10T01:37:27.5467492Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5467853Z 2025-10-10T01:37:27.5468468Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:27.5469025Z 2025-10-10T01:37:27.5469449Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:27.5469772Z 2025-10-10T01:37:27.5470374Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:27.5471164Z 2025-10-10T01:37:27.5471631Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:27.5471948Z 2025-10-10T01:37:27.5472254Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5472530Z 2025-10-10T01:37:27.5473039Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:27.5473502Z 2025-10-10T01:37:27.5473755Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] self._init_executor() 2025-10-10T01:37:27.5474015Z 2025-10-10T01:37:27.5474877Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:27.5475772Z 2025-10-10T01:37:27.5476344Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:27.5477308Z 2025-10-10T01:37:27.5478303Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:27.5479314Z 2025-10-10T01:37:27.5479903Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:27.5480263Z 2025-10-10T01:37:27.5480563Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5480845Z 2025-10-10T01:37:27.5481358Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:27.5481818Z 2025-10-10T01:37:27.5482088Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:37:27.5482365Z 2025-10-10T01:37:27.5482798Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5483068Z 2025-10-10T01:37:27.5483609Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:27.5484084Z 2025-10-10T01:37:27.5484371Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:27.5484665Z 2025-10-10T01:37:27.5484927Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5485202Z 2025-10-10T01:37:27.5485747Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:27.5486255Z 2025-10-10T01:37:27.5486563Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:37:27.5486873Z 2025-10-10T01:37:27.5487155Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5487441Z 2025-10-10T01:37:27.5487907Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:27.5488348Z 2025-10-10T01:37:27.5488702Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:27.5489047Z 2025-10-10T01:37:27.5489345Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5489638Z 2025-10-10T01:37:27.5489984Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:37:27.5490323Z 2025-10-10T01:37:27.5490675Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:37:27.5491022Z 2025-10-10T01:37:27.5491401Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:27.5491767Z 2025-10-10T01:37:27.5492106Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:37:27.5492441Z 2025-10-10T01:37:27.5492805Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:37:27.5493263Z 2025-10-10T01:37:27.5493648Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:27.5494009Z 2025-10-10T01:37:27.5494516Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:27.5494990Z 2025-10-10T01:37:27.5495350Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:27.5495693Z 2025-10-10T01:37:27.5496543Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:27.5497073Z 2025-10-10T01:37:27.5497459Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:27.5497830Z 2025-10-10T01:37:27.5498545Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:27.5499070Z 2025-10-10T01:37:27.5499367Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:37:27.5499681Z 2025-10-10T01:37:27.5500312Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:27.5500884Z 2025-10-10T01:37:27.5501252Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:27.5501606Z 2025-10-10T01:37:27.5501864Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5502127Z 2025-10-10T01:37:27.5502695Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:27.5503207Z 2025-10-10T01:37:27.5503499Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:37:27.5503795Z 2025-10-10T01:37:27.5504075Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5504370Z 2025-10-10T01:37:27.5504990Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:27.5505552Z 2025-10-10T01:37:27.5505836Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:37:27.5506124Z 2025-10-10T01:37:27.5506380Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5506647Z 2025-10-10T01:37:27.5507231Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:27.5507763Z 2025-10-10T01:37:27.5508084Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:27.5508396Z 2025-10-10T01:37:27.5508667Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5509077Z 2025-10-10T01:37:27.5509610Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:27.5510100Z 2025-10-10T01:37:27.5510381Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:37:27.5510674Z 2025-10-10T01:37:27.5510932Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5511202Z 2025-10-10T01:37:27.5511721Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:27.5512211Z 2025-10-10T01:37:27.5512518Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:37:27.5512822Z 2025-10-10T01:37:27.5513164Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:37:27.5513414Z 2025-10-10T01:37:27.5513905Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:27.5514361Z 2025-10-10T01:37:27.5514610Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] raise RuntimeError( 2025-10-10T01:37:27.5514882Z 2025-10-10T01:37:27.5515455Z (EngineCore_DP0 pid=1513) ERROR 10-10 01:37:27 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:27.5516154Z (EngineCore_DP0 pid=1513) Process EngineCore_DP0: 2025-10-10T01:37:27.5516568Z (EngineCore_DP0 pid=1513) Traceback (most recent call last): 2025-10-10T01:37:27.5517191Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:37:27.5517710Z (EngineCore_DP0 pid=1513) self.run() 2025-10-10T01:37:27.5518242Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:37:27.5518812Z (EngineCore_DP0 pid=1513) self._target(*self._args, **self._kwargs) 2025-10-10T01:37:27.5519611Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:37:27.5520174Z (EngineCore_DP0 pid=1513) raise e 2025-10-10T01:37:27.5520763Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:27.5521402Z (EngineCore_DP0 pid=1513) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:27.5521861Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5522489Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:27.5523129Z (EngineCore_DP0 pid=1513) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:27.5523801Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:27.5524419Z (EngineCore_DP0 pid=1513) self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:27.5524873Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5525523Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:27.5526220Z (EngineCore_DP0 pid=1513) self._init_executor() 2025-10-10T01:37:27.5526884Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:27.5527568Z (EngineCore_DP0 pid=1513) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:27.5541157Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:27.5541958Z (EngineCore_DP0 pid=1513) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:27.5542650Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5543352Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:27.5544107Z (EngineCore_DP0 pid=1513) return func(*args, **kwargs) 2025-10-10T01:37:27.5544534Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5545190Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:27.5545837Z (EngineCore_DP0 pid=1513) worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:27.5546270Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5546967Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:27.5547643Z (EngineCore_DP0 pid=1513) module = importlib.import_module(module_name) 2025-10-10T01:37:27.5548105Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5548706Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:27.5549350Z (EngineCore_DP0 pid=1513) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:27.5549850Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5550359Z (EngineCore_DP0 pid=1513) File "", line 1387, in _gcd_import 2025-10-10T01:37:27.5550927Z (EngineCore_DP0 pid=1513) File "", line 1360, in _find_and_load 2025-10-10T01:37:27.5551521Z (EngineCore_DP0 pid=1513) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:27.5552096Z (EngineCore_DP0 pid=1513) File "", line 935, in _load_unlocked 2025-10-10T01:37:27.5552674Z (EngineCore_DP0 pid=1513) File "", line 999, in exec_module 2025-10-10T01:37:27.5553278Z (EngineCore_DP0 pid=1513) File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:27.5554034Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:27.5554747Z (EngineCore_DP0 pid=1513) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:27.5555477Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:27.5556205Z (EngineCore_DP0 pid=1513) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:27.5557108Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:27.5557789Z (EngineCore_DP0 pid=1513) class FlashAttentionMetadataBuilder( 2025-10-10T01:37:27.5558589Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:27.5559488Z (EngineCore_DP0 pid=1513) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:27.5559970Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5560672Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:27.5561365Z (EngineCore_DP0 pid=1513) if not is_fa_version_supported(fa_version): 2025-10-10T01:37:27.5561816Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5562652Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:27.5563382Z (EngineCore_DP0 pid=1513) return _is_fa2_supported(device)[0] 2025-10-10T01:37:27.5563801Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5564518Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:27.5565237Z (EngineCore_DP0 pid=1513) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:27.5565695Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5566369Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:27.5567011Z (EngineCore_DP0 pid=1513) prop = get_device_properties(device) 2025-10-10T01:37:27.5567432Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:27.5568094Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:27.5568758Z (EngineCore_DP0 pid=1513) _lazy_init() # will define _get_device_properties 2025-10-10T01:37:27.5569164Z (EngineCore_DP0 pid=1513) ^^^^^^^^^^^^ 2025-10-10T01:37:27.5569755Z (EngineCore_DP0 pid=1513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:27.5570324Z (EngineCore_DP0 pid=1513) raise RuntimeError( 2025-10-10T01:37:27.5571019Z (EngineCore_DP0 pid=1513) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:27.9476498Z FAILED 2025-10-10T01:37:27.9604228Z models/test_initialization.py::test_can_initialize_small_subset[Gemma3nForCausalLM] Fork a new process to run a test 1517 2025-10-10T01:37:27.9614421Z Fork a new process to run a test 0 2025-10-10T01:37:27.9890476Z INFO 10-10 01:37:27 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma3nForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/gemma-3n-E2B-it'} 2025-10-10T01:37:28.1231911Z 2025-10-10T01:37:28.1233544Z config.json: 0% 0.00/4.25k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:37:40.1210618Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] EngineCore failed to start. 2025-10-10T01:37:40.1211188Z 2025-10-10T01:37:40.1211624Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] Traceback (most recent call last): 2025-10-10T01:37:40.1211945Z 2025-10-10T01:37:40.1212616Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:40.1213115Z 2025-10-10T01:37:40.1213563Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:40.1213909Z 2025-10-10T01:37:40.1214592Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1214949Z 2025-10-10T01:37:40.1215601Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:40.1216194Z 2025-10-10T01:37:40.1216679Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:40.1217117Z 2025-10-10T01:37:40.1217713Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:40.1218195Z 2025-10-10T01:37:40.1218621Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:40.1218957Z 2025-10-10T01:37:40.1219293Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1219687Z 2025-10-10T01:37:40.1220281Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:40.1220815Z 2025-10-10T01:37:40.1221129Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] self._init_executor() 2025-10-10T01:37:40.1221461Z 2025-10-10T01:37:40.1222151Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:40.1222659Z 2025-10-10T01:37:40.1223114Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:40.1223454Z 2025-10-10T01:37:40.1224125Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:40.1224740Z 2025-10-10T01:37:40.1225106Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:40.1225526Z 2025-10-10T01:37:40.1225859Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1226151Z 2025-10-10T01:37:40.1226771Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:40.1227338Z 2025-10-10T01:37:40.1227846Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:37:40.1228275Z 2025-10-10T01:37:40.1228563Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1228821Z 2025-10-10T01:37:40.1229508Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:40.1230115Z 2025-10-10T01:37:40.1230416Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:40.1230824Z 2025-10-10T01:37:40.1231123Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1231406Z 2025-10-10T01:37:40.1232057Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:40.1232670Z 2025-10-10T01:37:40.1233139Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:37:40.1233596Z 2025-10-10T01:37:40.1233900Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1234303Z 2025-10-10T01:37:40.1234778Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:40.1235325Z 2025-10-10T01:37:40.1235685Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:40.1236143Z 2025-10-10T01:37:40.1236446Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1236831Z 2025-10-10T01:37:40.1237213Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:37:40.1237609Z 2025-10-10T01:37:40.1238053Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:37:40.1238448Z 2025-10-10T01:37:40.1238925Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:40.1239534Z 2025-10-10T01:37:40.1239906Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:37:40.1240372Z 2025-10-10T01:37:40.1240750Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:37:40.1241203Z 2025-10-10T01:37:40.1241619Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:40.1242079Z 2025-10-10T01:37:40.1242613Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:40.1243194Z 2025-10-10T01:37:40.1243561Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:40.1244022Z 2025-10-10T01:37:40.1244635Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:40.1245167Z 2025-10-10T01:37:40.1245805Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:40.1246181Z 2025-10-10T01:37:40.1246882Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:40.1247517Z 2025-10-10T01:37:40.1247824Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:37:40.1248249Z 2025-10-10T01:37:40.1248984Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:40.1249591Z 2025-10-10T01:37:40.1250085Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:40.1250455Z 2025-10-10T01:37:40.1250971Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1251257Z 2025-10-10T01:37:40.1252001Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:40.1252639Z 2025-10-10T01:37:40.1252960Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:37:40.1253257Z 2025-10-10T01:37:40.1253653Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1253934Z 2025-10-10T01:37:40.1254543Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:40.1255228Z 2025-10-10T01:37:40.1255539Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:37:40.1255837Z 2025-10-10T01:37:40.1256097Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1256374Z 2025-10-10T01:37:40.1256975Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:40.1257640Z 2025-10-10T01:37:40.1257977Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:40.1258286Z 2025-10-10T01:37:40.1258565Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1258841Z 2025-10-10T01:37:40.1259370Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:40.1259852Z 2025-10-10T01:37:40.1260130Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:37:40.1260418Z 2025-10-10T01:37:40.1260677Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1260970Z 2025-10-10T01:37:40.1261614Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:40.1262107Z 2025-10-10T01:37:40.1262418Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:37:40.1262822Z 2025-10-10T01:37:40.1263053Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:37:40.1263296Z 2025-10-10T01:37:40.1263783Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:40.1264233Z 2025-10-10T01:37:40.1264484Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] raise RuntimeError( 2025-10-10T01:37:40.1264740Z 2025-10-10T01:37:40.1265315Z (EngineCore_DP0 pid=1596) ERROR 10-10 01:37:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:40.1266015Z (EngineCore_DP0 pid=1596) Process EngineCore_DP0: 2025-10-10T01:37:40.1266417Z (EngineCore_DP0 pid=1596) Traceback (most recent call last): 2025-10-10T01:37:40.1267139Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:37:40.1267665Z (EngineCore_DP0 pid=1596) self.run() 2025-10-10T01:37:40.1268332Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:37:40.1268920Z (EngineCore_DP0 pid=1596) self._target(*self._args, **self._kwargs) 2025-10-10T01:37:40.1269580Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:37:40.1270125Z (EngineCore_DP0 pid=1596) raise e 2025-10-10T01:37:40.1270722Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:40.1271354Z (EngineCore_DP0 pid=1596) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:40.1271798Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1272429Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:40.1273070Z (EngineCore_DP0 pid=1596) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:40.1273749Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:40.1274368Z (EngineCore_DP0 pid=1596) self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:40.1274823Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1275467Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:40.1276057Z (EngineCore_DP0 pid=1596) self._init_executor() 2025-10-10T01:37:40.1276722Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:40.1277407Z (EngineCore_DP0 pid=1596) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:40.1278129Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:40.1278838Z (EngineCore_DP0 pid=1596) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:40.1279398Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1280158Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:40.1280748Z (EngineCore_DP0 pid=1596) return func(*args, **kwargs) 2025-10-10T01:37:40.1281134Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1281926Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:40.1282572Z (EngineCore_DP0 pid=1596) worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:40.1283002Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1283666Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:40.1284322Z (EngineCore_DP0 pid=1596) module = importlib.import_module(module_name) 2025-10-10T01:37:40.1284857Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1285445Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:40.1286087Z (EngineCore_DP0 pid=1596) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:40.1286582Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1287081Z (EngineCore_DP0 pid=1596) File "", line 1387, in _gcd_import 2025-10-10T01:37:40.1287638Z (EngineCore_DP0 pid=1596) File "", line 1360, in _find_and_load 2025-10-10T01:37:40.1288216Z (EngineCore_DP0 pid=1596) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:40.1288803Z (EngineCore_DP0 pid=1596) File "", line 935, in _load_unlocked 2025-10-10T01:37:40.1289380Z (EngineCore_DP0 pid=1596) File "", line 999, in exec_module 2025-10-10T01:37:40.1289988Z (EngineCore_DP0 pid=1596) File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:40.1290723Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:40.1291412Z (EngineCore_DP0 pid=1596) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:40.1292160Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:40.1292896Z (EngineCore_DP0 pid=1596) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:40.1293687Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:40.1294360Z (EngineCore_DP0 pid=1596) class FlashAttentionMetadataBuilder( 2025-10-10T01:37:40.1295168Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:40.1295976Z (EngineCore_DP0 pid=1596) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:40.1296707Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1297408Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:40.1298237Z (EngineCore_DP0 pid=1596) if not is_fa_version_supported(fa_version): 2025-10-10T01:37:40.1298678Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1299436Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:40.1300135Z (EngineCore_DP0 pid=1596) return _is_fa2_supported(device)[0] 2025-10-10T01:37:40.1300553Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1301266Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:40.1301980Z (EngineCore_DP0 pid=1596) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:40.1302434Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1303230Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:40.1303878Z (EngineCore_DP0 pid=1596) prop = get_device_properties(device) 2025-10-10T01:37:40.1304295Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:40.1304982Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:40.1305667Z (EngineCore_DP0 pid=1596) _lazy_init() # will define _get_device_properties 2025-10-10T01:37:40.1306062Z (EngineCore_DP0 pid=1596) ^^^^^^^^^^^^ 2025-10-10T01:37:40.1306649Z (EngineCore_DP0 pid=1596) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:40.1307220Z (EngineCore_DP0 pid=1596) raise RuntimeError( 2025-10-10T01:37:40.1307953Z (EngineCore_DP0 pid=1596) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:40.5702120Z FAILED 2025-10-10T01:37:40.5830325Z models/test_initialization.py::test_can_initialize_small_subset[JinaVLForRanking] Fork a new process to run a test 1600 2025-10-10T01:37:40.5841244Z Fork a new process to run a test 0 2025-10-10T01:37:40.6123255Z INFO 10-10 01:37:40 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='JinaVLForRanking', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'jinaai/jina-reranker-m0'} 2025-10-10T01:37:40.7338788Z 2025-10-10T01:37:40.7341060Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:37:40.7341407Z config.json: 1.16kB [00:00, 5.82MB/s] 2025-10-10T01:37:40.8577443Z 2025-10-10T01:37:40.8578889Z preprocessor_config.json: 0% 0.00/316 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:37:49.5143034Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] EngineCore failed to start. 2025-10-10T01:37:49.5143919Z 2025-10-10T01:37:49.5144361Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] Traceback (most recent call last): 2025-10-10T01:37:49.5144737Z 2025-10-10T01:37:49.5145411Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:49.5146016Z 2025-10-10T01:37:49.5146404Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:49.5146783Z 2025-10-10T01:37:49.5147126Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5147470Z 2025-10-10T01:37:49.5148068Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:49.5148623Z 2025-10-10T01:37:49.5149221Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:49.5149644Z 2025-10-10T01:37:49.5150151Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:49.5150598Z 2025-10-10T01:37:49.5150917Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:49.5151224Z 2025-10-10T01:37:49.5151500Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5151777Z 2025-10-10T01:37:49.5152282Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:49.5152773Z 2025-10-10T01:37:49.5153029Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] self._init_executor() 2025-10-10T01:37:49.5153294Z 2025-10-10T01:37:49.5153826Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:49.5154332Z 2025-10-10T01:37:49.5154650Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:49.5154967Z 2025-10-10T01:37:49.5155502Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:49.5155994Z 2025-10-10T01:37:49.5156339Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:49.5156674Z 2025-10-10T01:37:49.5156970Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5157249Z 2025-10-10T01:37:49.5157730Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:49.5158183Z 2025-10-10T01:37:49.5158447Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:37:49.5158720Z 2025-10-10T01:37:49.5159088Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5159363Z 2025-10-10T01:37:49.5159878Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:49.5160441Z 2025-10-10T01:37:49.5160733Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:49.5161021Z 2025-10-10T01:37:49.5161289Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5161569Z 2025-10-10T01:37:49.5162096Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:49.5162573Z 2025-10-10T01:37:49.5162883Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:37:49.5163185Z 2025-10-10T01:37:49.5163457Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5163758Z 2025-10-10T01:37:49.5164279Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:49.5164701Z 2025-10-10T01:37:49.5165075Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:49.5165405Z 2025-10-10T01:37:49.5165695Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5165978Z 2025-10-10T01:37:49.5166322Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:37:49.5166659Z 2025-10-10T01:37:49.5167005Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:37:49.5167346Z 2025-10-10T01:37:49.5167722Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:49.5168088Z 2025-10-10T01:37:49.5168430Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:37:49.5168768Z 2025-10-10T01:37:49.5169136Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:37:49.5169488Z 2025-10-10T01:37:49.5169882Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:49.5170241Z 2025-10-10T01:37:49.5170757Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:49.5171222Z 2025-10-10T01:37:49.5171578Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:49.5171909Z 2025-10-10T01:37:49.5172432Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:49.5172911Z 2025-10-10T01:37:49.5173278Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:49.5173642Z 2025-10-10T01:37:49.5174188Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:49.5174776Z 2025-10-10T01:37:49.5175069Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:37:49.5175370Z 2025-10-10T01:37:49.5175992Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:49.5176566Z 2025-10-10T01:37:49.5176926Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:49.5177275Z 2025-10-10T01:37:49.5177534Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5177792Z 2025-10-10T01:37:49.5178355Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:49.5178873Z 2025-10-10T01:37:49.5179242Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:37:49.5179549Z 2025-10-10T01:37:49.5179826Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5180100Z 2025-10-10T01:37:49.5180698Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:49.5181253Z 2025-10-10T01:37:49.5181532Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:37:49.5181816Z 2025-10-10T01:37:49.5182071Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5182354Z 2025-10-10T01:37:49.5182968Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:49.5183506Z 2025-10-10T01:37:49.5183829Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:49.5184140Z 2025-10-10T01:37:49.5184419Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5184691Z 2025-10-10T01:37:49.5185216Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:49.5185699Z 2025-10-10T01:37:49.5185980Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:37:49.5186274Z 2025-10-10T01:37:49.5186550Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5186823Z 2025-10-10T01:37:49.5187342Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:49.5187824Z 2025-10-10T01:37:49.5188126Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:37:49.5188451Z 2025-10-10T01:37:49.5188675Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:37:49.5188924Z 2025-10-10T01:37:49.5189406Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:49.5189940Z 2025-10-10T01:37:49.5190188Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] raise RuntimeError( 2025-10-10T01:37:49.5190448Z 2025-10-10T01:37:49.5191020Z (EngineCore_DP0 pid=1680) ERROR 10-10 01:37:49 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:49.5191715Z (EngineCore_DP0 pid=1680) Process EngineCore_DP0: 2025-10-10T01:37:49.5192105Z (EngineCore_DP0 pid=1680) Traceback (most recent call last): 2025-10-10T01:37:49.5192709Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:37:49.5193230Z (EngineCore_DP0 pid=1680) self.run() 2025-10-10T01:37:49.5193756Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:37:49.5194322Z (EngineCore_DP0 pid=1680) self._target(*self._args, **self._kwargs) 2025-10-10T01:37:49.5195076Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:37:49.5195635Z (EngineCore_DP0 pid=1680) raise e 2025-10-10T01:37:49.5196606Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:49.5197320Z (EngineCore_DP0 pid=1680) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:49.5197771Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5198396Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:49.5199127Z (EngineCore_DP0 pid=1680) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:49.5199802Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:49.5200417Z (EngineCore_DP0 pid=1680) self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:49.5200879Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5201535Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:49.5202118Z (EngineCore_DP0 pid=1680) self._init_executor() 2025-10-10T01:37:49.5202782Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:49.5203473Z (EngineCore_DP0 pid=1680) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:49.5204198Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:49.5204914Z (EngineCore_DP0 pid=1680) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:49.5205412Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5206176Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:49.5206796Z (EngineCore_DP0 pid=1680) return func(*args, **kwargs) 2025-10-10T01:37:49.5207208Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5207878Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:49.5208683Z (EngineCore_DP0 pid=1680) worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:49.5209128Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5209805Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:49.5210464Z (EngineCore_DP0 pid=1680) module = importlib.import_module(module_name) 2025-10-10T01:37:49.5210914Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5211492Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:49.5212335Z (EngineCore_DP0 pid=1680) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:49.5213019Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5213667Z (EngineCore_DP0 pid=1680) File "", line 1387, in _gcd_import 2025-10-10T01:37:49.5214252Z (EngineCore_DP0 pid=1680) File "", line 1360, in _find_and_load 2025-10-10T01:37:49.5214839Z (EngineCore_DP0 pid=1680) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:49.5215424Z (EngineCore_DP0 pid=1680) File "", line 935, in _load_unlocked 2025-10-10T01:37:49.5215996Z (EngineCore_DP0 pid=1680) File "", line 999, in exec_module 2025-10-10T01:37:49.5216617Z (EngineCore_DP0 pid=1680) File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:49.5217364Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:49.5218055Z (EngineCore_DP0 pid=1680) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:49.5218778Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:49.5219514Z (EngineCore_DP0 pid=1680) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:49.5220302Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:49.5220974Z (EngineCore_DP0 pid=1680) class FlashAttentionMetadataBuilder( 2025-10-10T01:37:49.5221766Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:49.5222582Z (EngineCore_DP0 pid=1680) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:49.5223066Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5223761Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:49.5224449Z (EngineCore_DP0 pid=1680) if not is_fa_version_supported(fa_version): 2025-10-10T01:37:49.5224888Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5225630Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:49.5226416Z (EngineCore_DP0 pid=1680) return _is_fa2_supported(device)[0] 2025-10-10T01:37:49.5226845Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5227566Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:49.5228280Z (EngineCore_DP0 pid=1680) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:49.5228742Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5229410Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:49.5230044Z (EngineCore_DP0 pid=1680) prop = get_device_properties(device) 2025-10-10T01:37:49.5230457Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:49.5231188Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:49.5231868Z (EngineCore_DP0 pid=1680) _lazy_init() # will define _get_device_properties 2025-10-10T01:37:49.5232265Z (EngineCore_DP0 pid=1680) ^^^^^^^^^^^^ 2025-10-10T01:37:49.5232846Z (EngineCore_DP0 pid=1680) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:49.5233409Z (EngineCore_DP0 pid=1680) raise RuntimeError( 2025-10-10T01:37:49.5234091Z (EngineCore_DP0 pid=1680) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:49.9306990Z FAILED 2025-10-10T01:37:49.9436592Z models/test_initialization.py::test_can_initialize_small_subset[InternVLChatModel] Fork a new process to run a test 1684 2025-10-10T01:37:49.9446961Z Fork a new process to run a test 0 2025-10-10T01:37:49.9721522Z INFO 10-10 01:37:49 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternVLChatModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'OpenGVLab/InternVL2-1B'} 2025-10-10T01:37:50.0721732Z 2025-10-10T01:37:50.0723863Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:37:50.0724155Z config.json: 3.73kB [00:00, 18.0MB/s] 2025-10-10T01:37:50.1371978Z 2025-10-10T01:37:50.1373695Z configuration_internvl_chat.py: 0.00B [00:00, ?B/s] 2025-10-10T01:37:50.1374111Z configuration_internvl_chat.py: 4.04kB [00:00, 26.2MB/s] 2025-10-10T01:37:50.1883058Z 2025-10-10T01:37:50.1884941Z configuration_intern_vit.py: 0.00B [00:00, ?B/s] 2025-10-10T01:37:50.1885357Z configuration_intern_vit.py: 5.55kB [00:00, 41.0MB/s] 2025-10-10T01:37:50.1906401Z A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2-1B: 2025-10-10T01:37:50.1906917Z - configuration_intern_vit.py 2025-10-10T01:37:50.1907539Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:37:50.1908569Z A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2-1B: 2025-10-10T01:37:50.1909075Z - configuration_internvl_chat.py 2025-10-10T01:37:50.1909350Z - configuration_intern_vit.py 2025-10-10T01:37:50.1909942Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:37:50.4184590Z 2025-10-10T01:37:50.4185096Z preprocessor_config.json: 0% 0.00/287 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:37:58.3820582Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] EngineCore failed to start. 2025-10-10T01:37:58.3822262Z 2025-10-10T01:37:58.3822772Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] Traceback (most recent call last): 2025-10-10T01:37:58.3823171Z 2025-10-10T01:37:58.3823860Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:58.3824464Z 2025-10-10T01:37:58.3824862Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:58.3825265Z 2025-10-10T01:37:58.3825624Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3825985Z 2025-10-10T01:37:58.3826607Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:58.3827315Z 2025-10-10T01:37:58.3828064Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:58.3828450Z 2025-10-10T01:37:58.3828982Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:58.3829423Z 2025-10-10T01:37:58.3829749Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:58.3830059Z 2025-10-10T01:37:58.3830334Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3830614Z 2025-10-10T01:37:58.3831301Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:58.3832002Z 2025-10-10T01:37:58.3832508Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] self._init_executor() 2025-10-10T01:37:58.3832946Z 2025-10-10T01:37:58.3833900Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:58.3834704Z 2025-10-10T01:37:58.3835228Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:58.3835750Z 2025-10-10T01:37:58.3836438Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:58.3836960Z 2025-10-10T01:37:58.3837333Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:58.3837689Z 2025-10-10T01:37:58.3837999Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3838281Z 2025-10-10T01:37:58.3838779Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:58.3839373Z 2025-10-10T01:37:58.3839649Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:37:58.3839942Z 2025-10-10T01:37:58.3840195Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3840462Z 2025-10-10T01:37:58.3840985Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:58.3841629Z 2025-10-10T01:37:58.3841953Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:58.3842251Z 2025-10-10T01:37:58.3842527Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3842796Z 2025-10-10T01:37:58.3843350Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:58.3843843Z 2025-10-10T01:37:58.3844160Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:37:58.3844461Z 2025-10-10T01:37:58.3844731Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3845027Z 2025-10-10T01:37:58.3845555Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:58.3845987Z 2025-10-10T01:37:58.3846331Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:58.3846667Z 2025-10-10T01:37:58.3846954Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3847247Z 2025-10-10T01:37:58.3847588Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:37:58.3847918Z 2025-10-10T01:37:58.3848271Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:37:58.3848612Z 2025-10-10T01:37:58.3848996Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:58.3849356Z 2025-10-10T01:37:58.3849708Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:37:58.3850049Z 2025-10-10T01:37:58.3850411Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:37:58.3850768Z 2025-10-10T01:37:58.3851142Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:58.3851507Z 2025-10-10T01:37:58.3852005Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:58.3852487Z 2025-10-10T01:37:58.3852845Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:58.3853184Z 2025-10-10T01:37:58.3853703Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:58.3854181Z 2025-10-10T01:37:58.3854555Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:58.3854915Z 2025-10-10T01:37:58.3855473Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:58.3856065Z 2025-10-10T01:37:58.3856373Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:37:58.3856673Z 2025-10-10T01:37:58.3857318Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:58.3857892Z 2025-10-10T01:37:58.3858263Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:58.3858624Z 2025-10-10T01:37:58.3858880Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3859155Z 2025-10-10T01:37:58.3859722Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:58.3860250Z 2025-10-10T01:37:58.3860628Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:37:58.3860934Z 2025-10-10T01:37:58.3861214Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3861486Z 2025-10-10T01:37:58.3862091Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:58.3862639Z 2025-10-10T01:37:58.3862925Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:37:58.3863206Z 2025-10-10T01:37:58.3863472Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3863744Z 2025-10-10T01:37:58.3864327Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:58.3864870Z 2025-10-10T01:37:58.3865188Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:58.3865522Z 2025-10-10T01:37:58.3865792Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3866071Z 2025-10-10T01:37:58.3866593Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:58.3867078Z 2025-10-10T01:37:58.3867363Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:37:58.3867656Z 2025-10-10T01:37:58.3867930Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3868208Z 2025-10-10T01:37:58.3868734Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:58.3869218Z 2025-10-10T01:37:58.3869528Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:37:58.3869849Z 2025-10-10T01:37:58.3870075Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:37:58.3870324Z 2025-10-10T01:37:58.3870804Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:58.3871343Z 2025-10-10T01:37:58.3871594Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] raise RuntimeError( 2025-10-10T01:37:58.3871851Z 2025-10-10T01:37:58.3872432Z (EngineCore_DP0 pid=1742) ERROR 10-10 01:37:58 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:58.3873131Z (EngineCore_DP0 pid=1742) Process EngineCore_DP0: 2025-10-10T01:37:58.3873553Z (EngineCore_DP0 pid=1742) Traceback (most recent call last): 2025-10-10T01:37:58.3874162Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:37:58.3874673Z (EngineCore_DP0 pid=1742) self.run() 2025-10-10T01:37:58.3875201Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:37:58.3875850Z (EngineCore_DP0 pid=1742) self._target(*self._args, **self._kwargs) 2025-10-10T01:37:58.3876526Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:37:58.3877153Z (EngineCore_DP0 pid=1742) raise e 2025-10-10T01:37:58.3877842Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:37:58.3878619Z (EngineCore_DP0 pid=1742) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:37:58.3879181Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3879799Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:37:58.3880459Z (EngineCore_DP0 pid=1742) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:37:58.3881116Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:37:58.3881757Z (EngineCore_DP0 pid=1742) self.model_executor = executor_class(vllm_config) 2025-10-10T01:37:58.3882214Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3882863Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:37:58.3883449Z (EngineCore_DP0 pid=1742) self._init_executor() 2025-10-10T01:37:58.3884106Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:37:58.3884812Z (EngineCore_DP0 pid=1742) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:37:58.3885543Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:37:58.3886267Z (EngineCore_DP0 pid=1742) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:37:58.3886768Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3887416Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:37:58.3887997Z (EngineCore_DP0 pid=1742) return func(*args, **kwargs) 2025-10-10T01:37:58.3888384Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3889370Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:37:58.3890353Z (EngineCore_DP0 pid=1742) worker_class = resolve_obj_by_qualname( 2025-10-10T01:37:58.3890825Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3891513Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:37:58.3892194Z (EngineCore_DP0 pid=1742) module = importlib.import_module(module_name) 2025-10-10T01:37:58.3892647Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3893240Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:37:58.3893873Z (EngineCore_DP0 pid=1742) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:37:58.3894501Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3895018Z (EngineCore_DP0 pid=1742) File "", line 1387, in _gcd_import 2025-10-10T01:37:58.3895578Z (EngineCore_DP0 pid=1742) File "", line 1360, in _find_and_load 2025-10-10T01:37:58.3896420Z (EngineCore_DP0 pid=1742) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:37:58.3897061Z (EngineCore_DP0 pid=1742) File "", line 935, in _load_unlocked 2025-10-10T01:37:58.3897744Z (EngineCore_DP0 pid=1742) File "", line 999, in exec_module 2025-10-10T01:37:58.3898483Z (EngineCore_DP0 pid=1742) File "", line 488, in _call_with_frames_removed 2025-10-10T01:37:58.3899255Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:37:58.3899953Z (EngineCore_DP0 pid=1742) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:37:58.3900841Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:37:58.3901588Z (EngineCore_DP0 pid=1742) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:37:58.3902370Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:37:58.3903059Z (EngineCore_DP0 pid=1742) class FlashAttentionMetadataBuilder( 2025-10-10T01:37:58.3903856Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:37:58.3904688Z (EngineCore_DP0 pid=1742) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:37:58.3905171Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3905872Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:37:58.3906556Z (EngineCore_DP0 pid=1742) if not is_fa_version_supported(fa_version): 2025-10-10T01:37:58.3906998Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3907741Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:37:58.3908617Z (EngineCore_DP0 pid=1742) return _is_fa2_supported(device)[0] 2025-10-10T01:37:58.3909040Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3909767Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:37:58.3910485Z (EngineCore_DP0 pid=1742) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:37:58.3910939Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3911611Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:37:58.3912246Z (EngineCore_DP0 pid=1742) prop = get_device_properties(device) 2025-10-10T01:37:58.3912654Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:37:58.3913448Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:37:58.3914125Z (EngineCore_DP0 pid=1742) _lazy_init() # will define _get_device_properties 2025-10-10T01:37:58.3914521Z (EngineCore_DP0 pid=1742) ^^^^^^^^^^^^ 2025-10-10T01:37:58.3915104Z (EngineCore_DP0 pid=1742) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:37:58.3915672Z (EngineCore_DP0 pid=1742) raise RuntimeError( 2025-10-10T01:37:58.3916361Z (EngineCore_DP0 pid=1742) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:37:58.7918676Z FAILED 2025-10-10T01:37:58.8047602Z models/test_initialization.py::test_can_initialize_small_subset[InternLM2ForRewardModel] Fork a new process to run a test 1746 2025-10-10T01:37:58.8057650Z Fork a new process to run a test 0 2025-10-10T01:37:58.8333841Z INFO 10-10 01:37:58 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternLM2ForRewardModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'internlm/internlm2-1_8b-reward'} 2025-10-10T01:37:58.9141771Z 2025-10-10T01:37:58.9143564Z config.json: 0% 0.00/813 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:38:07.7758690Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] EngineCore failed to start. 2025-10-10T01:38:07.7759281Z 2025-10-10T01:38:07.7759879Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] Traceback (most recent call last): 2025-10-10T01:38:07.7760283Z 2025-10-10T01:38:07.7761112Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:38:07.7761927Z 2025-10-10T01:38:07.7762854Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:38:07.7763302Z 2025-10-10T01:38:07.7763679Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7764031Z 2025-10-10T01:38:07.7764652Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:38:07.7765215Z 2025-10-10T01:38:07.7765624Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:38:07.7766028Z 2025-10-10T01:38:07.7766576Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:38:07.7767037Z 2025-10-10T01:38:07.7767361Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:38:07.7767666Z 2025-10-10T01:38:07.7767955Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7768232Z 2025-10-10T01:38:07.7768830Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:38:07.7769523Z 2025-10-10T01:38:07.7770032Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] self._init_executor() 2025-10-10T01:38:07.7770493Z 2025-10-10T01:38:07.7771273Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:38:07.7772249Z 2025-10-10T01:38:07.7772702Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:38:07.7773034Z 2025-10-10T01:38:07.7773577Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:38:07.7774077Z 2025-10-10T01:38:07.7774420Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:38:07.7774753Z 2025-10-10T01:38:07.7775049Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7775339Z 2025-10-10T01:38:07.7776115Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:38:07.7776583Z 2025-10-10T01:38:07.7776862Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:38:07.7777129Z 2025-10-10T01:38:07.7777379Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7777644Z 2025-10-10T01:38:07.7778159Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:38:07.7778651Z 2025-10-10T01:38:07.7778954Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:38:07.7779253Z 2025-10-10T01:38:07.7779526Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7779795Z 2025-10-10T01:38:07.7780448Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:38:07.7780947Z 2025-10-10T01:38:07.7781259Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:38:07.7781556Z 2025-10-10T01:38:07.7781832Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7782101Z 2025-10-10T01:38:07.7782551Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:38:07.7782968Z 2025-10-10T01:38:07.7783310Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:38:07.7783644Z 2025-10-10T01:38:07.7783929Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7784210Z 2025-10-10T01:38:07.7784548Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:38:07.7784881Z 2025-10-10T01:38:07.7785224Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:38:07.7785562Z 2025-10-10T01:38:07.7785977Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:38:07.7786331Z 2025-10-10T01:38:07.7786675Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:38:07.7787026Z 2025-10-10T01:38:07.7787397Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:38:07.7787743Z 2025-10-10T01:38:07.7788114Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:38:07.7788475Z 2025-10-10T01:38:07.7788963Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:38:07.7789441Z 2025-10-10T01:38:07.7789786Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:38:07.7790214Z 2025-10-10T01:38:07.7790744Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:38:07.7791234Z 2025-10-10T01:38:07.7791627Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:38:07.7791998Z 2025-10-10T01:38:07.7792557Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:38:07.7793064Z 2025-10-10T01:38:07.7793364Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:38:07.7793659Z 2025-10-10T01:38:07.7794291Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:38:07.7794897Z 2025-10-10T01:38:07.7795357Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:38:07.7795720Z 2025-10-10T01:38:07.7795977Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7796463Z 2025-10-10T01:38:07.7797088Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:38:07.7797624Z 2025-10-10T01:38:07.7797932Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:38:07.7798229Z 2025-10-10T01:38:07.7798504Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7798802Z 2025-10-10T01:38:07.7799498Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:38:07.7800055Z 2025-10-10T01:38:07.7800340Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:38:07.7800632Z 2025-10-10T01:38:07.7800896Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7801181Z 2025-10-10T01:38:07.7801764Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:38:07.7802293Z 2025-10-10T01:38:07.7802624Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:38:07.7802941Z 2025-10-10T01:38:07.7803218Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7803492Z 2025-10-10T01:38:07.7804021Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:38:07.7804507Z 2025-10-10T01:38:07.7804785Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:38:07.7805087Z 2025-10-10T01:38:07.7805347Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7805619Z 2025-10-10T01:38:07.7806146Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:38:07.7806787Z 2025-10-10T01:38:07.7807098Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:38:07.7807403Z 2025-10-10T01:38:07.7807627Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:38:07.7807870Z 2025-10-10T01:38:07.7808361Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:38:07.7808808Z 2025-10-10T01:38:07.7809056Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] raise RuntimeError( 2025-10-10T01:38:07.7809310Z 2025-10-10T01:38:07.7809884Z (EngineCore_DP0 pid=1825) ERROR 10-10 01:38:07 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:38:07.7810704Z (EngineCore_DP0 pid=1825) Process EngineCore_DP0: 2025-10-10T01:38:07.7811126Z (EngineCore_DP0 pid=1825) Traceback (most recent call last): 2025-10-10T01:38:07.7811745Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:38:07.7812259Z (EngineCore_DP0 pid=1825) self.run() 2025-10-10T01:38:07.7812785Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:38:07.7813353Z (EngineCore_DP0 pid=1825) self._target(*self._args, **self._kwargs) 2025-10-10T01:38:07.7814026Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:38:07.7814592Z (EngineCore_DP0 pid=1825) raise e 2025-10-10T01:38:07.7815181Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:38:07.7826601Z (EngineCore_DP0 pid=1825) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:38:07.7827099Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7827763Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:38:07.7828445Z (EngineCore_DP0 pid=1825) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:38:07.7829117Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:38:07.7829753Z (EngineCore_DP0 pid=1825) self.model_executor = executor_class(vllm_config) 2025-10-10T01:38:07.7830213Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7830885Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:38:07.7831483Z (EngineCore_DP0 pid=1825) self._init_executor() 2025-10-10T01:38:07.7832167Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:38:07.7832863Z (EngineCore_DP0 pid=1825) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:38:07.7833594Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:38:07.7834473Z (EngineCore_DP0 pid=1825) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:38:07.7834985Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7835636Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:38:07.7836237Z (EngineCore_DP0 pid=1825) return func(*args, **kwargs) 2025-10-10T01:38:07.7836636Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7837275Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:38:07.7837905Z (EngineCore_DP0 pid=1825) worker_class = resolve_obj_by_qualname( 2025-10-10T01:38:07.7838338Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7839207Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:38:07.7839886Z (EngineCore_DP0 pid=1825) module = importlib.import_module(module_name) 2025-10-10T01:38:07.7840327Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7840911Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:38:07.7841541Z (EngineCore_DP0 pid=1825) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:38:07.7842037Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7842535Z (EngineCore_DP0 pid=1825) File "", line 1387, in _gcd_import 2025-10-10T01:38:07.7843092Z (EngineCore_DP0 pid=1825) File "", line 1360, in _find_and_load 2025-10-10T01:38:07.7843691Z (EngineCore_DP0 pid=1825) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:38:07.7844278Z (EngineCore_DP0 pid=1825) File "", line 935, in _load_unlocked 2025-10-10T01:38:07.7844863Z (EngineCore_DP0 pid=1825) File "", line 999, in exec_module 2025-10-10T01:38:07.7845484Z (EngineCore_DP0 pid=1825) File "", line 488, in _call_with_frames_removed 2025-10-10T01:38:07.7846224Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:38:07.7846928Z (EngineCore_DP0 pid=1825) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:38:07.7847665Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:38:07.7848409Z (EngineCore_DP0 pid=1825) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:38:07.7849199Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:38:07.7849872Z (EngineCore_DP0 pid=1825) class FlashAttentionMetadataBuilder( 2025-10-10T01:38:07.7850671Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:38:07.7851485Z (EngineCore_DP0 pid=1825) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:38:07.7851968Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7852767Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:38:07.7853456Z (EngineCore_DP0 pid=1825) if not is_fa_version_supported(fa_version): 2025-10-10T01:38:07.7853890Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7854635Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:38:07.7855334Z (EngineCore_DP0 pid=1825) return _is_fa2_supported(device)[0] 2025-10-10T01:38:07.7855751Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7856467Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:38:07.7857255Z (EngineCore_DP0 pid=1825) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:38:07.7857738Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7858414Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:38:07.7859054Z (EngineCore_DP0 pid=1825) prop = get_device_properties(device) 2025-10-10T01:38:07.7859474Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:07.7860132Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:38:07.7860806Z (EngineCore_DP0 pid=1825) _lazy_init() # will define _get_device_properties 2025-10-10T01:38:07.7861210Z (EngineCore_DP0 pid=1825) ^^^^^^^^^^^^ 2025-10-10T01:38:07.7861813Z (EngineCore_DP0 pid=1825) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:38:07.7862381Z (EngineCore_DP0 pid=1825) raise RuntimeError( 2025-10-10T01:38:07.7863060Z (EngineCore_DP0 pid=1825) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:38:08.1842379Z FAILED 2025-10-10T01:38:08.1971027Z models/test_initialization.py::test_can_initialize_small_subset[TransformersForMultimodalLM] Fork a new process to run a test 1829 2025-10-10T01:38:08.1981421Z Fork a new process to run a test 0 2025-10-10T01:38:08.2247469Z INFO 10-10 01:38:08 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='TransformersForMultimodalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'transformers', 'model': 'BAAI/Emu3-Chat-hf'} 2025-10-10T01:38:08.4048756Z 2025-10-10T01:38:08.4262614Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:38:08.4263101Z config.json: 6.55MB [00:00, 307MB/s] 2025-10-10T01:38:09.0462285Z 2025-10-10T01:38:09.0462855Z preprocessor_config.json: 0% 0.00/611 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:38:38.2682607Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] EngineCore failed to start. 2025-10-10T01:38:38.2683072Z 2025-10-10T01:38:38.2683464Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] Traceback (most recent call last): 2025-10-10T01:38:38.2683820Z 2025-10-10T01:38:38.2684486Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:38:38.2685337Z 2025-10-10T01:38:38.2685680Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:38:38.2685979Z 2025-10-10T01:38:38.2686259Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2686535Z 2025-10-10T01:38:38.2687023Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:38:38.2687473Z 2025-10-10T01:38:38.2687800Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:38:38.2688126Z 2025-10-10T01:38:38.2688792Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:38:38.2689253Z 2025-10-10T01:38:38.2689800Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:38:38.2690120Z 2025-10-10T01:38:38.2690405Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2690679Z 2025-10-10T01:38:38.2691206Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:38:38.2691690Z 2025-10-10T01:38:38.2692006Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] self._init_executor() 2025-10-10T01:38:38.2692498Z 2025-10-10T01:38:38.2693439Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:38:38.2694088Z 2025-10-10T01:38:38.2694607Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:38:38.2695070Z 2025-10-10T01:38:38.2695728Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:38:38.2696666Z 2025-10-10T01:38:38.2697050Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:38:38.2697499Z 2025-10-10T01:38:38.2697798Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2698084Z 2025-10-10T01:38:38.2698571Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:38:38.2699044Z 2025-10-10T01:38:38.2699309Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:38:38.2699575Z 2025-10-10T01:38:38.2699824Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2700078Z 2025-10-10T01:38:38.2700592Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:38:38.2701080Z 2025-10-10T01:38:38.2701388Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:38:38.2701673Z 2025-10-10T01:38:38.2701934Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2702412Z 2025-10-10T01:38:38.2702976Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:38:38.2703476Z 2025-10-10T01:38:38.2703779Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:38:38.2704090Z 2025-10-10T01:38:38.2704362Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2704635Z 2025-10-10T01:38:38.2705088Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:38:38.2705496Z 2025-10-10T01:38:38.2705841Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:38:38.2706170Z 2025-10-10T01:38:38.2706577Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2706856Z 2025-10-10T01:38:38.2707207Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:38:38.2707535Z 2025-10-10T01:38:38.2707876Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:38:38.2708214Z 2025-10-10T01:38:38.2708598Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:38:38.2708962Z 2025-10-10T01:38:38.2709302Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:38:38.2709640Z 2025-10-10T01:38:38.2709999Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:38:38.2710345Z 2025-10-10T01:38:38.2710723Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:38:38.2711075Z 2025-10-10T01:38:38.2711571Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:38:38.2712039Z 2025-10-10T01:38:38.2712393Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:38:38.2712731Z 2025-10-10T01:38:38.2713251Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:38:38.2713736Z 2025-10-10T01:38:38.2714103Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:38:38.2714464Z 2025-10-10T01:38:38.2715008Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:38:38.2715517Z 2025-10-10T01:38:38.2715806Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:38:38.2716111Z 2025-10-10T01:38:38.2716729Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:38:38.2717386Z 2025-10-10T01:38:38.2717753Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:38:38.2718099Z 2025-10-10T01:38:38.2718358Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2718617Z 2025-10-10T01:38:38.2719328Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:38:38.2719841Z 2025-10-10T01:38:38.2720144Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:38:38.2720435Z 2025-10-10T01:38:38.2720709Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2720984Z 2025-10-10T01:38:38.2721661Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:38:38.2722216Z 2025-10-10T01:38:38.2722499Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:38:38.2722785Z 2025-10-10T01:38:38.2723041Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2723307Z 2025-10-10T01:38:38.2723885Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:38:38.2724413Z 2025-10-10T01:38:38.2724725Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:38:38.2725034Z 2025-10-10T01:38:38.2725313Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2725583Z 2025-10-10T01:38:38.2726106Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:38:38.2726580Z 2025-10-10T01:38:38.2726877Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:38:38.2727158Z 2025-10-10T01:38:38.2727418Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2727687Z 2025-10-10T01:38:38.2728199Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:38:38.2728683Z 2025-10-10T01:38:38.2728991Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:38:38.2729297Z 2025-10-10T01:38:38.2729522Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:38:38.2729762Z 2025-10-10T01:38:38.2730243Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:38:38.2730688Z 2025-10-10T01:38:38.2730937Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] raise RuntimeError( 2025-10-10T01:38:38.2731192Z 2025-10-10T01:38:38.2731767Z (EngineCore_DP0 pid=1909) ERROR 10-10 01:38:38 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:38:38.2732582Z (EngineCore_DP0 pid=1909) Process EngineCore_DP0: 2025-10-10T01:38:38.2732980Z (EngineCore_DP0 pid=1909) Traceback (most recent call last): 2025-10-10T01:38:38.2733586Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:38:38.2734097Z (EngineCore_DP0 pid=1909) self.run() 2025-10-10T01:38:38.2734618Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:38:38.2735171Z (EngineCore_DP0 pid=1909) self._target(*self._args, **self._kwargs) 2025-10-10T01:38:38.2735833Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:38:38.2736371Z (EngineCore_DP0 pid=1909) raise e 2025-10-10T01:38:38.2737055Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:38:38.2737688Z (EngineCore_DP0 pid=1909) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:38:38.2738121Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2738732Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:38:38.2739357Z (EngineCore_DP0 pid=1909) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:38:38.2740008Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:38:38.2740631Z (EngineCore_DP0 pid=1909) self.model_executor = executor_class(vllm_config) 2025-10-10T01:38:38.2741092Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2741736Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:38:38.2742311Z (EngineCore_DP0 pid=1909) self._init_executor() 2025-10-10T01:38:38.2742972Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:38:38.2743671Z (EngineCore_DP0 pid=1909) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:38:38.2744391Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:38:38.2745100Z (EngineCore_DP0 pid=1909) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:38:38.2745592Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2746230Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:38:38.2746812Z (EngineCore_DP0 pid=1909) return func(*args, **kwargs) 2025-10-10T01:38:38.2747191Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2747821Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:38:38.2748456Z (EngineCore_DP0 pid=1909) worker_class = resolve_obj_by_qualname( 2025-10-10T01:38:38.2748875Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2749543Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:38:38.2750284Z (EngineCore_DP0 pid=1909) module = importlib.import_module(module_name) 2025-10-10T01:38:38.2750725Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2751292Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:38:38.2751913Z (EngineCore_DP0 pid=1909) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:38:38.2752403Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2752905Z (EngineCore_DP0 pid=1909) File "", line 1387, in _gcd_import 2025-10-10T01:38:38.2753458Z (EngineCore_DP0 pid=1909) File "", line 1360, in _find_and_load 2025-10-10T01:38:38.2754146Z (EngineCore_DP0 pid=1909) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:38:38.2754722Z (EngineCore_DP0 pid=1909) File "", line 935, in _load_unlocked 2025-10-10T01:38:38.2755288Z (EngineCore_DP0 pid=1909) File "", line 999, in exec_module 2025-10-10T01:38:38.2755888Z (EngineCore_DP0 pid=1909) File "", line 488, in _call_with_frames_removed 2025-10-10T01:38:38.2756616Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:38:38.2757289Z (EngineCore_DP0 pid=1909) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:38:38.2757994Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:38:38.2758714Z (EngineCore_DP0 pid=1909) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:38:38.2759557Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:38:38.2760226Z (EngineCore_DP0 pid=1909) class FlashAttentionMetadataBuilder( 2025-10-10T01:38:38.2761004Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:38:38.2761823Z (EngineCore_DP0 pid=1909) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:38:38.2762289Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2762979Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:38:38.2763655Z (EngineCore_DP0 pid=1909) if not is_fa_version_supported(fa_version): 2025-10-10T01:38:38.2764083Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2764820Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:38:38.2765503Z (EngineCore_DP0 pid=1909) return _is_fa2_supported(device)[0] 2025-10-10T01:38:38.2765908Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2766609Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:38:38.2767415Z (EngineCore_DP0 pid=1909) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:38:38.2767864Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2768523Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:38:38.2769151Z (EngineCore_DP0 pid=1909) prop = get_device_properties(device) 2025-10-10T01:38:38.2769567Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:38.2770215Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:38:38.2770866Z (EngineCore_DP0 pid=1909) _lazy_init() # will define _get_device_properties 2025-10-10T01:38:38.2771256Z (EngineCore_DP0 pid=1909) ^^^^^^^^^^^^ 2025-10-10T01:38:38.2771939Z (EngineCore_DP0 pid=1909) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:38:38.2772508Z (EngineCore_DP0 pid=1909) raise RuntimeError( 2025-10-10T01:38:38.2773190Z (EngineCore_DP0 pid=1909) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:38:38.7000446Z FAILED 2025-10-10T01:38:38.7129154Z models/test_initialization.py::test_can_initialize_small_subset[PrithviGeoSpatialMAE] Fork a new process to run a test 1913 2025-10-10T01:38:38.7140523Z Fork a new process to run a test 0 2025-10-10T01:38:38.7420324Z INFO 10-10 01:38:38 [utils.py:233] non-default args: {'skip_tokenizer_init': True, 'load_format': 'dummy', 'dtype': torch.float16, 'gpu_memory_utilization': 0.8, 'max_num_seqs': 32, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='PrithviGeoSpatialMAE', exist_overrides={}, use_original_num_layers=False), 'enforce_eager': True, 'model_impl': 'vllm', 'model': 'ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11'} 2025-10-10T01:38:38.8887797Z 2025-10-10T01:38:38.8889771Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:38:38.8890113Z config.json: 3.45kB [00:00, 21.2MB/s] 2025-10-10T01:38:50.0159275Z INFO 10-10 01:38:50 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:38:50.0159968Z INFO 10-10 01:38:50 [model.py:551] Resolved architecture: Terratorch 2025-10-10T01:38:50.0160374Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:38:50.0829091Z ERROR 10-10 01:38:50 [config.py:278] Error retrieving safetensors: 'ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files., retrying 1 of 2 2025-10-10T01:38:52.1378629Z ERROR 10-10 01:38:52 [config.py:276] Error retrieving safetensors: 'ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files. 2025-10-10T01:38:52.1379703Z INFO 10-10 01:38:52 [model.py:1765] Downcasting torch.float32 to torch.float16. 2025-10-10T01:38:52.1381048Z WARNING 10-10 01:38:52 [model.py:1867] The model's config.json does not contain any of the following keys to determine the original maximum length of the model: ['max_position_embeddings', 'n_positions', 'max_seq_len', 'seq_length', 'model_max_length', 'max_target_positions', 'max_sequence_length', 'max_seq_length', 'seq_len']. Assuming the model's maximum length is 2048. 2025-10-10T01:38:52.1382303Z INFO 10-10 01:38:52 [model.py:1545] Using max model len 2048 2025-10-10T01:38:52.2127956Z INFO 10-10 01:38:52 [arg_utils.py:1580] (Disabling) chunked prefill by default 2025-10-10T01:38:52.2128457Z INFO 10-10 01:38:52 [arg_utils.py:1583] (Disabling) prefix caching by default 2025-10-10T01:38:52.3615920Z INFO 10-10 01:38:52 [vllm.py:345] Cudagraph is disabled under eager mode 2025-10-10T01:38:52.3616546Z INFO 10-10 01:38:52 [vllm.py:404] Only "last" pooling supports chunked prefill and prefix caching; disabling both. 2025-10-10T01:38:55.8301874Z (EngineCore_DP0 pid=1974) INFO 10-10 01:38:55 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:38:55.8384684Z (EngineCore_DP0 pid=1974) INFO 10-10 01:38:55 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11', speculative_config=None, tokenizer='ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11', skip_tokenizer_init=True, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11, enable_prefix_caching=False, chunked_prefill_enabled=False, pooler_config=PoolerConfig(pooling_type='All', normalize=None, dimensions=None, enable_chunked_processing=None, max_embed_len=None, activation=None, logit_bias=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={'level': 0, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 0, 'local_cache_dir': None} 2025-10-10T01:38:55.9586043Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] EngineCore failed to start. 2025-10-10T01:38:55.9586447Z 2025-10-10T01:38:55.9586815Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] Traceback (most recent call last): 2025-10-10T01:38:55.9587195Z 2025-10-10T01:38:55.9587863Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:38:55.9588456Z 2025-10-10T01:38:55.9588837Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:38:55.9589252Z 2025-10-10T01:38:55.9589614Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9589992Z 2025-10-10T01:38:55.9590493Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:38:55.9590935Z 2025-10-10T01:38:55.9591270Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:38:55.9591596Z 2025-10-10T01:38:55.9592078Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:38:55.9592511Z 2025-10-10T01:38:55.9592827Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:38:55.9593521Z 2025-10-10T01:38:55.9593857Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9594135Z 2025-10-10T01:38:55.9594645Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:38:55.9595106Z 2025-10-10T01:38:55.9595360Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] self._init_executor() 2025-10-10T01:38:55.9595619Z 2025-10-10T01:38:55.9596475Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:38:55.9597398Z 2025-10-10T01:38:55.9598027Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:38:55.9598655Z 2025-10-10T01:38:55.9599876Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:38:55.9600422Z 2025-10-10T01:38:55.9600809Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:38:55.9601148Z 2025-10-10T01:38:55.9601439Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9601719Z 2025-10-10T01:38:55.9602213Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:38:55.9602680Z 2025-10-10T01:38:55.9602949Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:38:55.9603219Z 2025-10-10T01:38:55.9603471Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9603724Z 2025-10-10T01:38:55.9604256Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:38:55.9604751Z 2025-10-10T01:38:55.9605045Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:38:55.9605340Z 2025-10-10T01:38:55.9605608Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9605891Z 2025-10-10T01:38:55.9606424Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:38:55.9606931Z 2025-10-10T01:38:55.9607243Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:38:55.9607546Z 2025-10-10T01:38:55.9607822Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9608096Z 2025-10-10T01:38:55.9608552Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:38:55.9608964Z 2025-10-10T01:38:55.9609315Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:38:55.9609647Z 2025-10-10T01:38:55.9609937Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9610359Z 2025-10-10T01:38:55.9610707Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:38:55.9611039Z 2025-10-10T01:38:55.9611383Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:38:55.9611720Z 2025-10-10T01:38:55.9612090Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:38:55.9612453Z 2025-10-10T01:38:55.9612792Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:38:55.9613119Z 2025-10-10T01:38:55.9613495Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:38:55.9613853Z 2025-10-10T01:38:55.9614326Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:38:55.9614684Z 2025-10-10T01:38:55.9615188Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:38:55.9615675Z 2025-10-10T01:38:55.9616024Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:38:55.9616362Z 2025-10-10T01:38:55.9616878Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:38:55.9617364Z 2025-10-10T01:38:55.9617738Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:38:55.9618100Z 2025-10-10T01:38:55.9618647Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:38:55.9619157Z 2025-10-10T01:38:55.9619447Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:38:55.9619737Z 2025-10-10T01:38:55.9620364Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:38:55.9620932Z 2025-10-10T01:38:55.9621299Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:38:55.9621652Z 2025-10-10T01:38:55.9621916Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9622177Z 2025-10-10T01:38:55.9622747Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:38:55.9623268Z 2025-10-10T01:38:55.9623563Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:38:55.9623863Z 2025-10-10T01:38:55.9624132Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9624405Z 2025-10-10T01:38:55.9625002Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:38:55.9625643Z 2025-10-10T01:38:55.9625924Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:38:55.9626210Z 2025-10-10T01:38:55.9626467Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9626729Z 2025-10-10T01:38:55.9627318Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:38:55.9627841Z 2025-10-10T01:38:55.9628165Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:38:55.9628475Z 2025-10-10T01:38:55.9628751Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9629025Z 2025-10-10T01:38:55.9629628Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:38:55.9630118Z 2025-10-10T01:38:55.9630398Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:38:55.9630684Z 2025-10-10T01:38:55.9630945Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9631216Z 2025-10-10T01:38:55.9631735Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:38:55.9632220Z 2025-10-10T01:38:55.9632529Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:38:55.9632839Z 2025-10-10T01:38:55.9633084Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:38:55.9633325Z 2025-10-10T01:38:55.9633813Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:38:55.9634258Z 2025-10-10T01:38:55.9634506Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] raise RuntimeError( 2025-10-10T01:38:55.9634759Z 2025-10-10T01:38:55.9635317Z (EngineCore_DP0 pid=1974) ERROR 10-10 01:38:55 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:38:55.9636011Z (EngineCore_DP0 pid=1974) Process EngineCore_DP0: 2025-10-10T01:38:55.9636407Z (EngineCore_DP0 pid=1974) Traceback (most recent call last): 2025-10-10T01:38:55.9637070Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:38:55.9637599Z (EngineCore_DP0 pid=1974) self.run() 2025-10-10T01:38:55.9638131Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:38:55.9638694Z (EngineCore_DP0 pid=1974) self._target(*self._args, **self._kwargs) 2025-10-10T01:38:55.9639476Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:38:55.9640029Z (EngineCore_DP0 pid=1974) raise e 2025-10-10T01:38:55.9640615Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:38:55.9641422Z (EngineCore_DP0 pid=1974) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:38:55.9641863Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9642476Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:38:55.9643114Z (EngineCore_DP0 pid=1974) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:38:55.9643772Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:38:55.9644402Z (EngineCore_DP0 pid=1974) self.model_executor = executor_class(vllm_config) 2025-10-10T01:38:55.9644882Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9645647Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:38:55.9646237Z (EngineCore_DP0 pid=1974) self._init_executor() 2025-10-10T01:38:55.9646903Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:38:55.9647593Z (EngineCore_DP0 pid=1974) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:38:55.9648317Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:38:55.9649028Z (EngineCore_DP0 pid=1974) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:38:55.9649540Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9650197Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:38:55.9650788Z (EngineCore_DP0 pid=1974) return func(*args, **kwargs) 2025-10-10T01:38:55.9651172Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9651808Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:38:55.9652435Z (EngineCore_DP0 pid=1974) worker_class = resolve_obj_by_qualname( 2025-10-10T01:38:55.9652851Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9653515Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:38:55.9654173Z (EngineCore_DP0 pid=1974) module = importlib.import_module(module_name) 2025-10-10T01:38:55.9654618Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9655194Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:38:55.9655807Z (EngineCore_DP0 pid=1974) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:38:55.9656294Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9656785Z (EngineCore_DP0 pid=1974) File "", line 1387, in _gcd_import 2025-10-10T01:38:55.9657341Z (EngineCore_DP0 pid=1974) File "", line 1360, in _find_and_load 2025-10-10T01:38:55.9657929Z (EngineCore_DP0 pid=1974) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:38:55.9658594Z (EngineCore_DP0 pid=1974) File "", line 935, in _load_unlocked 2025-10-10T01:38:55.9659162Z (EngineCore_DP0 pid=1974) File "", line 999, in exec_module 2025-10-10T01:38:55.9659758Z (EngineCore_DP0 pid=1974) File "", line 488, in _call_with_frames_removed 2025-10-10T01:38:55.9660500Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:38:55.9661178Z (EngineCore_DP0 pid=1974) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:38:55.9661896Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:38:55.9662613Z (EngineCore_DP0 pid=1974) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:38:55.9663473Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:38:55.9664151Z (EngineCore_DP0 pid=1974) class FlashAttentionMetadataBuilder( 2025-10-10T01:38:55.9664940Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:38:55.9665766Z (EngineCore_DP0 pid=1974) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:38:55.9666241Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9666929Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:38:55.9667616Z (EngineCore_DP0 pid=1974) if not is_fa_version_supported(fa_version): 2025-10-10T01:38:55.9668058Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9668805Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:38:55.9669500Z (EngineCore_DP0 pid=1974) return _is_fa2_supported(device)[0] 2025-10-10T01:38:55.9669921Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9670637Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:38:55.9671352Z (EngineCore_DP0 pid=1974) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:38:55.9671809Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9672481Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:38:55.9673122Z (EngineCore_DP0 pid=1974) prop = get_device_properties(device) 2025-10-10T01:38:55.9673545Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:38:55.9674199Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:38:55.9674856Z (EngineCore_DP0 pid=1974) _lazy_init() # will define _get_device_properties 2025-10-10T01:38:55.9675249Z (EngineCore_DP0 pid=1974) ^^^^^^^^^^^^ 2025-10-10T01:38:55.9675823Z (EngineCore_DP0 pid=1974) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:38:55.9676471Z (EngineCore_DP0 pid=1974) raise RuntimeError( 2025-10-10T01:38:55.9677166Z (EngineCore_DP0 pid=1974) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:38:56.4285692Z FAILED 2025-10-10T01:38:56.4414281Z models/test_initialization.py::test_can_initialize_small_subset[UltravoxModel] Fork a new process to run a test 1978 2025-10-10T01:38:56.4423507Z Fork a new process to run a test 0 2025-10-10T01:38:56.4694999Z INFO 10-10 01:38:56 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='UltravoxModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'fixie-ai/ultravox-v0_5-llama-3_2-1b'} 2025-10-10T01:38:56.6579474Z 2025-10-10T01:38:56.6581577Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:38:56.6582214Z config.json: 2.15kB [00:00, 11.6MB/s] 2025-10-10T01:38:56.8904358Z 2025-10-10T01:38:56.8905283Z preprocessor_config.json: 0% 0.00/339 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:39:05.3967825Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] EngineCore failed to start. 2025-10-10T01:39:05.3968255Z 2025-10-10T01:39:05.3968668Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] Traceback (most recent call last): 2025-10-10T01:39:05.3968983Z 2025-10-10T01:39:05.3969531Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:05.3970014Z 2025-10-10T01:39:05.3970322Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:05.3970628Z 2025-10-10T01:39:05.3970901Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3971179Z 2025-10-10T01:39:05.3971713Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:05.3972178Z 2025-10-10T01:39:05.3972503Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:05.3972817Z 2025-10-10T01:39:05.3973291Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:05.3973725Z 2025-10-10T01:39:05.3980086Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:05.3980698Z 2025-10-10T01:39:05.3981244Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3981612Z 2025-10-10T01:39:05.3982371Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:05.3983058Z 2025-10-10T01:39:05.3983349Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] self._init_executor() 2025-10-10T01:39:05.3983633Z 2025-10-10T01:39:05.3984200Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:05.3984709Z 2025-10-10T01:39:05.3985056Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:05.3985374Z 2025-10-10T01:39:05.3985912Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:05.3986547Z 2025-10-10T01:39:05.3986905Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:05.3987244Z 2025-10-10T01:39:05.3987532Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3987820Z 2025-10-10T01:39:05.3988308Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:05.3988783Z 2025-10-10T01:39:05.3989052Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:39:05.3989326Z 2025-10-10T01:39:05.3989575Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3989839Z 2025-10-10T01:39:05.3990478Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:05.3990960Z 2025-10-10T01:39:05.3991269Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:05.3991556Z 2025-10-10T01:39:05.3991826Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3992093Z 2025-10-10T01:39:05.3992623Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:05.3993104Z 2025-10-10T01:39:05.3993404Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:39:05.3993709Z 2025-10-10T01:39:05.3993984Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3994261Z 2025-10-10T01:39:05.3994729Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:05.3995137Z 2025-10-10T01:39:05.3995479Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:05.3995807Z 2025-10-10T01:39:05.3996418Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.3996719Z 2025-10-10T01:39:05.3997068Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:39:05.3997407Z 2025-10-10T01:39:05.3997755Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:39:05.3998098Z 2025-10-10T01:39:05.3998480Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:05.3998840Z 2025-10-10T01:39:05.3999314Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:39:05.3999648Z 2025-10-10T01:39:05.4000019Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:39:05.4000370Z 2025-10-10T01:39:05.4000750Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:05.4001220Z 2025-10-10T01:39:05.4001735Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:05.4002195Z 2025-10-10T01:39:05.4002543Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:05.4002890Z 2025-10-10T01:39:05.4003409Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:05.4003891Z 2025-10-10T01:39:05.4004259Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:05.4004618Z 2025-10-10T01:39:05.4005284Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:05.4005817Z 2025-10-10T01:39:05.4006116Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:39:05.4006413Z 2025-10-10T01:39:05.4007033Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:05.4007601Z 2025-10-10T01:39:05.4007966Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:05.4008315Z 2025-10-10T01:39:05.4008573Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4008841Z 2025-10-10T01:39:05.4009404Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:05.4009925Z 2025-10-10T01:39:05.4010219Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:39:05.4010513Z 2025-10-10T01:39:05.4010779Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4011051Z 2025-10-10T01:39:05.4011642Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:05.4012303Z 2025-10-10T01:39:05.4012585Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:39:05.4012873Z 2025-10-10T01:39:05.4013132Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4013396Z 2025-10-10T01:39:05.4013990Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:05.4014512Z 2025-10-10T01:39:05.4014829Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:05.4015136Z 2025-10-10T01:39:05.4015411Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4015678Z 2025-10-10T01:39:05.4016196Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:05.4016722Z 2025-10-10T01:39:05.4017004Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:39:05.4017289Z 2025-10-10T01:39:05.4017545Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4017813Z 2025-10-10T01:39:05.4018326Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:05.4018806Z 2025-10-10T01:39:05.4019111Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:39:05.4019409Z 2025-10-10T01:39:05.4019650Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:39:05.4019888Z 2025-10-10T01:39:05.4020443Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:05.4020887Z 2025-10-10T01:39:05.4021135Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] raise RuntimeError( 2025-10-10T01:39:05.4021385Z 2025-10-10T01:39:05.4021944Z (EngineCore_DP0 pid=2056) ERROR 10-10 01:39:05 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:05.4022631Z (EngineCore_DP0 pid=2056) Process EngineCore_DP0: 2025-10-10T01:39:05.4023023Z (EngineCore_DP0 pid=2056) Traceback (most recent call last): 2025-10-10T01:39:05.4023624Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:39:05.4024135Z (EngineCore_DP0 pid=2056) self.run() 2025-10-10T01:39:05.4024652Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:39:05.4025206Z (EngineCore_DP0 pid=2056) self._target(*self._args, **self._kwargs) 2025-10-10T01:39:05.4025864Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:39:05.4026408Z (EngineCore_DP0 pid=2056) raise e 2025-10-10T01:39:05.4026990Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:05.4027666Z (EngineCore_DP0 pid=2056) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:05.4028103Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4028722Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:05.4029358Z (EngineCore_DP0 pid=2056) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:05.4030011Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:05.4030621Z (EngineCore_DP0 pid=2056) self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:05.4031061Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4031714Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:05.4032309Z (EngineCore_DP0 pid=2056) self._init_executor() 2025-10-10T01:39:05.4033023Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:05.4033702Z (EngineCore_DP0 pid=2056) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:05.4034414Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:05.4035123Z (EngineCore_DP0 pid=2056) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:05.4035639Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4036273Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:05.4036866Z (EngineCore_DP0 pid=2056) return func(*args, **kwargs) 2025-10-10T01:39:05.4037259Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4037996Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:05.4038645Z (EngineCore_DP0 pid=2056) worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:05.4039069Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4039802Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:05.4040458Z (EngineCore_DP0 pid=2056) module = importlib.import_module(module_name) 2025-10-10T01:39:05.4040902Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4041482Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:05.4042101Z (EngineCore_DP0 pid=2056) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:05.4042593Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4043083Z (EngineCore_DP0 pid=2056) File "", line 1387, in _gcd_import 2025-10-10T01:39:05.4043631Z (EngineCore_DP0 pid=2056) File "", line 1360, in _find_and_load 2025-10-10T01:39:05.4044229Z (EngineCore_DP0 pid=2056) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:05.4044851Z (EngineCore_DP0 pid=2056) File "", line 935, in _load_unlocked 2025-10-10T01:39:05.4045421Z (EngineCore_DP0 pid=2056) File "", line 999, in exec_module 2025-10-10T01:39:05.4046024Z (EngineCore_DP0 pid=2056) File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:05.4046751Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:05.4047425Z (EngineCore_DP0 pid=2056) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:05.4048147Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:05.4048864Z (EngineCore_DP0 pid=2056) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:05.4049642Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:05.4050367Z (EngineCore_DP0 pid=2056) class FlashAttentionMetadataBuilder( 2025-10-10T01:39:05.4051151Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:05.4051956Z (EngineCore_DP0 pid=2056) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:05.4052425Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4053108Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:05.4053786Z (EngineCore_DP0 pid=2056) if not is_fa_version_supported(fa_version): 2025-10-10T01:39:05.4054220Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4055044Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:05.4055747Z (EngineCore_DP0 pid=2056) return _is_fa2_supported(device)[0] 2025-10-10T01:39:05.4056150Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4056862Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:05.4057569Z (EngineCore_DP0 pid=2056) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:05.4058020Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4058684Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:05.4059312Z (EngineCore_DP0 pid=2056) prop = get_device_properties(device) 2025-10-10T01:39:05.4059742Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:05.4060402Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:05.4061063Z (EngineCore_DP0 pid=2056) _lazy_init() # will define _get_device_properties 2025-10-10T01:39:05.4061455Z (EngineCore_DP0 pid=2056) ^^^^^^^^^^^^ 2025-10-10T01:39:05.4062026Z (EngineCore_DP0 pid=2056) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:05.4062649Z (EngineCore_DP0 pid=2056) raise RuntimeError( 2025-10-10T01:39:05.4063327Z (EngineCore_DP0 pid=2056) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:05.8211754Z FAILED 2025-10-10T01:39:05.8340838Z models/test_initialization.py::test_can_initialize_small_subset[DeepSeekMTPModel] Fork a new process to run a test 2060 2025-10-10T01:39:05.8350337Z Fork a new process to run a test 0 2025-10-10T01:39:05.8623041Z INFO 10-10 01:39:05 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DeepSeekMTPModel', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'luccafong/deepseek_mtp_draft_random', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'luccafong/deepseek_mtp_main_random'} 2025-10-10T01:39:05.9735570Z 2025-10-10T01:39:05.9737541Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:39:05.9737881Z config.json: 1.95kB [00:00, 8.97MB/s] 2025-10-10T01:39:06.0245381Z INFO 10-10 01:39:06 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:39:12.6244387Z INFO 10-10 01:39:12 [model.py:551] Resolved architecture: DeepseekV3ForCausalLM 2025-10-10T01:39:12.6244949Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:39:12.6482713Z INFO 10-10 01:39:12 [model.py:1545] Using max model len 163840 2025-10-10T01:39:12.7480848Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:39:13.0613097Z 2025-10-10T01:39:13.0613513Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:39:13.0613943Z config.json: 1.95kB [00:00, 15.1MB/s] 2025-10-10T01:39:13.0941929Z INFO 10-10 01:39:13 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:39:20.1410227Z INFO 10-10 01:39:20 [model.py:551] Resolved architecture: DeepSeekMTPModel 2025-10-10T01:39:20.1411065Z INFO 10-10 01:39:20 [model.py:1545] Using max model len 163840 2025-10-10T01:39:20.1418508Z INFO 10-10 01:39:20 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:39:20.2998014Z 2025-10-10T01:39:20.3000095Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:39:20.3000413Z tokenizer_config.json: 3.13kB [00:00, 19.5MB/s] 2025-10-10T01:39:20.4339557Z 2025-10-10T01:39:20.4628018Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:39:20.4628350Z tokenizer.json: 7.85MB [00:00, 273MB/s] 2025-10-10T01:39:21.0607653Z INFO 10-10 01:39:21 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:39:21.0807130Z (EngineCore_DP0 pid=2168) INFO 10-10 01:39:21 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:39:21.0884684Z (EngineCore_DP0 pid=2168) INFO 10-10 01:39:21 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='luccafong/deepseek_mtp_main_random', speculative_config=SpeculativeConfig(method='mtp', model='luccafong/deepseek_mtp_draft_random', num_spec_tokens=1), tokenizer='luccafong/deepseek_mtp_main_random', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=163840, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=luccafong/deepseek_mtp_main_random, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': ['+quant_fp8'], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:39:21.2183318Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] EngineCore failed to start. 2025-10-10T01:39:21.2183986Z 2025-10-10T01:39:21.2184404Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] Traceback (most recent call last): 2025-10-10T01:39:21.2184784Z 2025-10-10T01:39:21.2185453Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:21.2186053Z 2025-10-10T01:39:21.2186448Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:21.2186829Z 2025-10-10T01:39:21.2187180Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2187706Z 2025-10-10T01:39:21.2188359Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:21.2188919Z 2025-10-10T01:39:21.2189563Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:21.2189975Z 2025-10-10T01:39:21.2190635Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:21.2191666Z 2025-10-10T01:39:21.2192397Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:21.2192985Z 2025-10-10T01:39:21.2193440Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2193888Z 2025-10-10T01:39:21.2194619Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:21.2195302Z 2025-10-10T01:39:21.2195585Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] self._init_executor() 2025-10-10T01:39:21.2195850Z 2025-10-10T01:39:21.2196620Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:21.2197149Z 2025-10-10T01:39:21.2197477Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:21.2197943Z 2025-10-10T01:39:21.2198483Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:21.2198978Z 2025-10-10T01:39:21.2199463Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:21.2199801Z 2025-10-10T01:39:21.2200096Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2200380Z 2025-10-10T01:39:21.2200891Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:21.2201350Z 2025-10-10T01:39:21.2201616Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:39:21.2201883Z 2025-10-10T01:39:21.2202133Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2202396Z 2025-10-10T01:39:21.2202907Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:21.2203484Z 2025-10-10T01:39:21.2203773Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:21.2204060Z 2025-10-10T01:39:21.2204335Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2204609Z 2025-10-10T01:39:21.2205130Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:21.2205612Z 2025-10-10T01:39:21.2205918Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:39:21.2206234Z 2025-10-10T01:39:21.2217265Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2217647Z 2025-10-10T01:39:21.2218400Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:21.2218830Z 2025-10-10T01:39:21.2219207Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:21.2219541Z 2025-10-10T01:39:21.2219838Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2220132Z 2025-10-10T01:39:21.2220477Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:39:21.2220816Z 2025-10-10T01:39:21.2221165Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:39:21.2221526Z 2025-10-10T01:39:21.2221928Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:21.2222313Z 2025-10-10T01:39:21.2222662Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:39:21.2223004Z 2025-10-10T01:39:21.2223379Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:39:21.2223792Z 2025-10-10T01:39:21.2224174Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:21.2224533Z 2025-10-10T01:39:21.2225046Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:21.2225515Z 2025-10-10T01:39:21.2225866Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:21.2226229Z 2025-10-10T01:39:21.2226762Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:21.2227249Z 2025-10-10T01:39:21.2227622Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:21.2227993Z 2025-10-10T01:39:21.2228544Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:21.2229141Z 2025-10-10T01:39:21.2229449Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:39:21.2229745Z 2025-10-10T01:39:21.2230383Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:21.2230956Z 2025-10-10T01:39:21.2231327Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:21.2231682Z 2025-10-10T01:39:21.2231945Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2232219Z 2025-10-10T01:39:21.2232785Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:21.2233311Z 2025-10-10T01:39:21.2233693Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:39:21.2234009Z 2025-10-10T01:39:21.2234284Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2234559Z 2025-10-10T01:39:21.2235317Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:21.2236002Z 2025-10-10T01:39:21.2236304Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:39:21.2236593Z 2025-10-10T01:39:21.2236851Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2237119Z 2025-10-10T01:39:21.2237705Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:21.2238227Z 2025-10-10T01:39:21.2238551Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:21.2238858Z 2025-10-10T01:39:21.2239227Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2239574Z 2025-10-10T01:39:21.2240104Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:21.2240590Z 2025-10-10T01:39:21.2240871Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:39:21.2241160Z 2025-10-10T01:39:21.2241424Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2241697Z 2025-10-10T01:39:21.2242213Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:21.2242696Z 2025-10-10T01:39:21.2243011Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:39:21.2243318Z 2025-10-10T01:39:21.2243570Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:39:21.2243809Z 2025-10-10T01:39:21.2244297Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:21.2244799Z 2025-10-10T01:39:21.2245056Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] raise RuntimeError( 2025-10-10T01:39:21.2245308Z 2025-10-10T01:39:21.2245873Z (EngineCore_DP0 pid=2168) ERROR 10-10 01:39:21 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:21.2246565Z (EngineCore_DP0 pid=2168) Process EngineCore_DP0: 2025-10-10T01:39:21.2246963Z (EngineCore_DP0 pid=2168) Traceback (most recent call last): 2025-10-10T01:39:21.2247576Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:39:21.2248091Z (EngineCore_DP0 pid=2168) self.run() 2025-10-10T01:39:21.2248622Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:39:21.2249283Z (EngineCore_DP0 pid=2168) self._target(*self._args, **self._kwargs) 2025-10-10T01:39:21.2249958Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:39:21.2250524Z (EngineCore_DP0 pid=2168) raise e 2025-10-10T01:39:21.2251116Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:21.2251743Z (EngineCore_DP0 pid=2168) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:21.2252230Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2252870Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:21.2253522Z (EngineCore_DP0 pid=2168) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:21.2254184Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:21.2254808Z (EngineCore_DP0 pid=2168) self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:21.2255267Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2255935Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:21.2256592Z (EngineCore_DP0 pid=2168) self._init_executor() 2025-10-10T01:39:21.2257261Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:21.2257955Z (EngineCore_DP0 pid=2168) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:21.2258683Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:21.2259395Z (EngineCore_DP0 pid=2168) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:21.2259888Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2260531Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:21.2261123Z (EngineCore_DP0 pid=2168) return func(*args, **kwargs) 2025-10-10T01:39:21.2261500Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2262208Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:21.2262833Z (EngineCore_DP0 pid=2168) worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:21.2263257Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2263924Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:21.2264572Z (EngineCore_DP0 pid=2168) module = importlib.import_module(module_name) 2025-10-10T01:39:21.2265017Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2265598Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:21.2266226Z (EngineCore_DP0 pid=2168) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:21.2266800Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2267302Z (EngineCore_DP0 pid=2168) File "", line 1387, in _gcd_import 2025-10-10T01:39:21.2267860Z (EngineCore_DP0 pid=2168) File "", line 1360, in _find_and_load 2025-10-10T01:39:21.2268446Z (EngineCore_DP0 pid=2168) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:21.2269026Z (EngineCore_DP0 pid=2168) File "", line 935, in _load_unlocked 2025-10-10T01:39:21.2269598Z (EngineCore_DP0 pid=2168) File "", line 999, in exec_module 2025-10-10T01:39:21.2270190Z (EngineCore_DP0 pid=2168) File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:21.2270928Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:21.2271606Z (EngineCore_DP0 pid=2168) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:21.2272318Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:21.2273044Z (EngineCore_DP0 pid=2168) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:21.2273898Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:21.2274579Z (EngineCore_DP0 pid=2168) class FlashAttentionMetadataBuilder( 2025-10-10T01:39:21.2275373Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:21.2276192Z (EngineCore_DP0 pid=2168) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:21.2276673Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2277362Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:21.2278039Z (EngineCore_DP0 pid=2168) if not is_fa_version_supported(fa_version): 2025-10-10T01:39:21.2278483Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2279311Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:21.2280095Z (EngineCore_DP0 pid=2168) return _is_fa2_supported(device)[0] 2025-10-10T01:39:21.2280519Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2281247Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:21.2281972Z (EngineCore_DP0 pid=2168) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:21.2282428Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2283110Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:21.2283750Z (EngineCore_DP0 pid=2168) prop = get_device_properties(device) 2025-10-10T01:39:21.2284172Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:21.2284896Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:21.2285566Z (EngineCore_DP0 pid=2168) _lazy_init() # will define _get_device_properties 2025-10-10T01:39:21.2285963Z (EngineCore_DP0 pid=2168) ^^^^^^^^^^^^ 2025-10-10T01:39:21.2286545Z (EngineCore_DP0 pid=2168) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:21.2287116Z (EngineCore_DP0 pid=2168) raise RuntimeError( 2025-10-10T01:39:21.2287807Z (EngineCore_DP0 pid=2168) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:21.6234092Z FAILED 2025-10-10T01:39:21.6363914Z models/test_initialization.py::test_can_initialize_small_subset[XLMRobertaModel] Fork a new process to run a test 2172 2025-10-10T01:39:21.6374613Z Fork a new process to run a test 0 2025-10-10T01:39:21.6649387Z INFO 10-10 01:39:21 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='XLMRobertaModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'intfloat/multilingual-e5-small'} 2025-10-10T01:39:21.7705369Z 2025-10-10T01:39:21.7706216Z config.json: 0% 0.00/655 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:39:30.9467332Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:39:30.9468032Z 2025-10-10T01:39:30.9468530Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:39:30.9469005Z 2025-10-10T01:39:30.9469849Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:30.9470588Z 2025-10-10T01:39:30.9470998Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:30.9471535Z 2025-10-10T01:39:30.9471908Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9472301Z 2025-10-10T01:39:30.9472808Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:30.9473507Z 2025-10-10T01:39:30.9474389Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:30.9474850Z 2025-10-10T01:39:30.9475419Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:30.9476010Z 2025-10-10T01:39:30.9476335Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:30.9476852Z 2025-10-10T01:39:30.9477422Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9477954Z 2025-10-10T01:39:30.9478985Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:30.9479987Z 2025-10-10T01:39:30.9480437Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] self._init_executor() 2025-10-10T01:39:30.9480828Z 2025-10-10T01:39:30.9481544Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:30.9482083Z 2025-10-10T01:39:30.9482433Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:30.9482955Z 2025-10-10T01:39:30.9483516Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:30.9484035Z 2025-10-10T01:39:30.9484390Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:30.9484724Z 2025-10-10T01:39:30.9485016Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9485307Z 2025-10-10T01:39:30.9485800Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:30.9486249Z 2025-10-10T01:39:30.9486509Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:39:30.9486783Z 2025-10-10T01:39:30.9487028Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9487296Z 2025-10-10T01:39:30.9487816Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:30.9488416Z 2025-10-10T01:39:30.9488710Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:30.9488997Z 2025-10-10T01:39:30.9489265Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9489529Z 2025-10-10T01:39:30.9490061Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:30.9490541Z 2025-10-10T01:39:30.9490847Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:39:30.9491143Z 2025-10-10T01:39:30.9491421Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9491835Z 2025-10-10T01:39:30.9492314Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:30.9492732Z 2025-10-10T01:39:30.9493069Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:30.9493401Z 2025-10-10T01:39:30.9493698Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9493982Z 2025-10-10T01:39:30.9494322Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:39:30.9494646Z 2025-10-10T01:39:30.9494994Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:39:30.9495331Z 2025-10-10T01:39:30.9495706Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:30.9496059Z 2025-10-10T01:39:30.9496594Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:39:30.9496925Z 2025-10-10T01:39:30.9497290Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:39:30.9497744Z 2025-10-10T01:39:30.9498120Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:30.9498483Z 2025-10-10T01:39:30.9498983Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:30.9499447Z 2025-10-10T01:39:30.9499795Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:30.9500144Z 2025-10-10T01:39:30.9500680Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:30.9501164Z 2025-10-10T01:39:30.9501534Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:30.9501892Z 2025-10-10T01:39:30.9502448Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:30.9503021Z 2025-10-10T01:39:30.9503324Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:39:30.9503614Z 2025-10-10T01:39:30.9504244Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:30.9504827Z 2025-10-10T01:39:30.9505195Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:30.9505544Z 2025-10-10T01:39:30.9505796Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9506075Z 2025-10-10T01:39:30.9506632Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:30.9507301Z 2025-10-10T01:39:30.9507608Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:39:30.9507907Z 2025-10-10T01:39:30.9508178Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9508445Z 2025-10-10T01:39:30.9509049Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:30.9509594Z 2025-10-10T01:39:30.9509874Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:39:30.9510155Z 2025-10-10T01:39:30.9510418Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9510685Z 2025-10-10T01:39:30.9511263Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:30.9511801Z 2025-10-10T01:39:30.9512115Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:30.9512428Z 2025-10-10T01:39:30.9512696Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9513025Z 2025-10-10T01:39:30.9513543Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:30.9514022Z 2025-10-10T01:39:30.9514304Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:39:30.9514588Z 2025-10-10T01:39:30.9514850Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9515113Z 2025-10-10T01:39:30.9515644Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:30.9516121Z 2025-10-10T01:39:30.9516432Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:39:30.9516731Z 2025-10-10T01:39:30.9516957Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:39:30.9517203Z 2025-10-10T01:39:30.9517680Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:30.9518181Z 2025-10-10T01:39:30.9518429Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] raise RuntimeError( 2025-10-10T01:39:30.9518687Z 2025-10-10T01:39:30.9519366Z (EngineCore_DP0 pid=2252) ERROR 10-10 01:39:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:30.9520057Z (EngineCore_DP0 pid=2252) Process EngineCore_DP0: 2025-10-10T01:39:30.9520448Z (EngineCore_DP0 pid=2252) Traceback (most recent call last): 2025-10-10T01:39:30.9521052Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:39:30.9521558Z (EngineCore_DP0 pid=2252) self.run() 2025-10-10T01:39:30.9522093Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:39:30.9522736Z (EngineCore_DP0 pid=2252) self._target(*self._args, **self._kwargs) 2025-10-10T01:39:30.9523419Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:39:30.9523961Z (EngineCore_DP0 pid=2252) raise e 2025-10-10T01:39:30.9524538Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:30.9525170Z (EngineCore_DP0 pid=2252) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:30.9525608Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9526221Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:30.9526861Z (EngineCore_DP0 pid=2252) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:30.9527509Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:30.9528128Z (EngineCore_DP0 pid=2252) self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:30.9528577Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9529224Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:30.9529900Z (EngineCore_DP0 pid=2252) self._init_executor() 2025-10-10T01:39:30.9530556Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:30.9531241Z (EngineCore_DP0 pid=2252) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:30.9531962Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:30.9532680Z (EngineCore_DP0 pid=2252) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:30.9533170Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9533802Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:30.9534394Z (EngineCore_DP0 pid=2252) return func(*args, **kwargs) 2025-10-10T01:39:30.9534778Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9535464Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:30.9536095Z (EngineCore_DP0 pid=2252) worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:30.9536515Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9537177Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:30.9537831Z (EngineCore_DP0 pid=2252) module = importlib.import_module(module_name) 2025-10-10T01:39:30.9538278Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9538855Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:30.9539483Z (EngineCore_DP0 pid=2252) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:30.9540044Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9540556Z (EngineCore_DP0 pid=2252) File "", line 1387, in _gcd_import 2025-10-10T01:39:30.9541105Z (EngineCore_DP0 pid=2252) File "", line 1360, in _find_and_load 2025-10-10T01:39:30.9541686Z (EngineCore_DP0 pid=2252) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:30.9542263Z (EngineCore_DP0 pid=2252) File "", line 935, in _load_unlocked 2025-10-10T01:39:30.9542826Z (EngineCore_DP0 pid=2252) File "", line 999, in exec_module 2025-10-10T01:39:30.9543419Z (EngineCore_DP0 pid=2252) File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:30.9544151Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:30.9544828Z (EngineCore_DP0 pid=2252) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:30.9545545Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:30.9546257Z (EngineCore_DP0 pid=2252) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:30.9547087Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:30.9547750Z (EngineCore_DP0 pid=2252) class FlashAttentionMetadataBuilder( 2025-10-10T01:39:30.9548533Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:30.9549340Z (EngineCore_DP0 pid=2252) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:30.9549815Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9550495Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:30.9551169Z (EngineCore_DP0 pid=2252) if not is_fa_version_supported(fa_version): 2025-10-10T01:39:30.9551625Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9552364Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:30.9553114Z (EngineCore_DP0 pid=2252) return _is_fa2_supported(device)[0] 2025-10-10T01:39:30.9553516Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9554234Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:30.9554949Z (EngineCore_DP0 pid=2252) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:30.9555399Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9556065Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:30.9556688Z (EngineCore_DP0 pid=2252) prop = get_device_properties(device) 2025-10-10T01:39:30.9557109Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:30.9557837Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:30.9558504Z (EngineCore_DP0 pid=2252) _lazy_init() # will define _get_device_properties 2025-10-10T01:39:30.9558912Z (EngineCore_DP0 pid=2252) ^^^^^^^^^^^^ 2025-10-10T01:39:30.9559528Z (EngineCore_DP0 pid=2252) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:30.9560086Z (EngineCore_DP0 pid=2252) raise RuntimeError( 2025-10-10T01:39:30.9560764Z (EngineCore_DP0 pid=2252) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:31.3773288Z FAILED 2025-10-10T01:39:31.3901314Z models/test_initialization.py::test_can_initialize_large_subset[Olmo2ForCausalLM] Fork a new process to run a test 2256 2025-10-10T01:39:31.3911296Z Fork a new process to run a test 0 2025-10-10T01:39:31.4179749Z INFO 10-10 01:39:31 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Olmo2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'allenai/OLMo-2-0425-1B'} 2025-10-10T01:39:31.5475324Z 2025-10-10T01:39:31.5476672Z config.json: 0% 0.00/623 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:39:39.2886438Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] EngineCore failed to start. 2025-10-10T01:39:39.2887129Z 2025-10-10T01:39:39.2887557Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] Traceback (most recent call last): 2025-10-10T01:39:39.2887926Z 2025-10-10T01:39:39.2888593Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:39.2889222Z 2025-10-10T01:39:39.2889621Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:39.2889946Z 2025-10-10T01:39:39.2890233Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2890504Z 2025-10-10T01:39:39.2890998Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:39.2891443Z 2025-10-10T01:39:39.2891989Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:39.2892389Z 2025-10-10T01:39:39.2892883Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:39.2893462Z 2025-10-10T01:39:39.2893776Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:39.2894087Z 2025-10-10T01:39:39.2894361Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2894636Z 2025-10-10T01:39:39.2895139Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:39.2895602Z 2025-10-10T01:39:39.2896300Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] self._init_executor() 2025-10-10T01:39:39.2896748Z 2025-10-10T01:39:39.2898001Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:39.2898837Z 2025-10-10T01:39:39.2899271Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:39.2899777Z 2025-10-10T01:39:39.2900345Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:39.2900859Z 2025-10-10T01:39:39.2901200Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:39.2901538Z 2025-10-10T01:39:39.2901822Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2902105Z 2025-10-10T01:39:39.2902595Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:39.2903047Z 2025-10-10T01:39:39.2903308Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:39:39.2903573Z 2025-10-10T01:39:39.2903824Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2904077Z 2025-10-10T01:39:39.2904594Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:39.2905182Z 2025-10-10T01:39:39.2905475Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:39.2905761Z 2025-10-10T01:39:39.2906027Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2906295Z 2025-10-10T01:39:39.2906822Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:39.2907321Z 2025-10-10T01:39:39.2907621Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:39:39.2907923Z 2025-10-10T01:39:39.2908191Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2908472Z 2025-10-10T01:39:39.2908906Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:39.2909400Z 2025-10-10T01:39:39.2909746Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:39.2910074Z 2025-10-10T01:39:39.2910361Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2910634Z 2025-10-10T01:39:39.2910985Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:39:39.2911318Z 2025-10-10T01:39:39.2911663Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:39:39.2912022Z 2025-10-10T01:39:39.2912398Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:39.2912753Z 2025-10-10T01:39:39.2913196Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:39:39.2913530Z 2025-10-10T01:39:39.2913894Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:39:39.2914249Z 2025-10-10T01:39:39.2914620Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:39.2914984Z 2025-10-10T01:39:39.2915476Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:39.2915943Z 2025-10-10T01:39:39.2916292Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:39.2916626Z 2025-10-10T01:39:39.2917144Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:39.2917634Z 2025-10-10T01:39:39.2918020Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:39.2918389Z 2025-10-10T01:39:39.2918959Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:39.2919664Z 2025-10-10T01:39:39.2919971Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:39:39.2920263Z 2025-10-10T01:39:39.2920895Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:39.2921470Z 2025-10-10T01:39:39.2921835Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:39.2922189Z 2025-10-10T01:39:39.2922441Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2922704Z 2025-10-10T01:39:39.2923262Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:39.2923781Z 2025-10-10T01:39:39.2924073Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:39:39.2924367Z 2025-10-10T01:39:39.2924718Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2924994Z 2025-10-10T01:39:39.2925613Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:39.2926165Z 2025-10-10T01:39:39.2926449Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:39:39.2926728Z 2025-10-10T01:39:39.2926988Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2927251Z 2025-10-10T01:39:39.2927824Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:39.2928353Z 2025-10-10T01:39:39.2928746Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:39.2929059Z 2025-10-10T01:39:39.2929329Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2929614Z 2025-10-10T01:39:39.2930133Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:39.2930614Z 2025-10-10T01:39:39.2930888Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:39:39.2931169Z 2025-10-10T01:39:39.2931430Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2931696Z 2025-10-10T01:39:39.2932217Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:39.2932706Z 2025-10-10T01:39:39.2933024Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:39:39.2933323Z 2025-10-10T01:39:39.2933546Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:39:39.2933789Z 2025-10-10T01:39:39.2934264Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:39.2934760Z 2025-10-10T01:39:39.2935004Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] raise RuntimeError( 2025-10-10T01:39:39.2935262Z 2025-10-10T01:39:39.2935831Z (EngineCore_DP0 pid=2314) ERROR 10-10 01:39:39 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:39.2936527Z (EngineCore_DP0 pid=2314) Process EngineCore_DP0: 2025-10-10T01:39:39.2936940Z (EngineCore_DP0 pid=2314) Traceback (most recent call last): 2025-10-10T01:39:39.2937542Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:39:39.2938068Z (EngineCore_DP0 pid=2314) self.run() 2025-10-10T01:39:39.2938587Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:39:39.2939148Z (EngineCore_DP0 pid=2314) self._target(*self._args, **self._kwargs) 2025-10-10T01:39:39.2939819Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:39:39.2940419Z (EngineCore_DP0 pid=2314) raise e 2025-10-10T01:39:39.2941012Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:39.2941643Z (EngineCore_DP0 pid=2314) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:39.2942113Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2942752Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:39.2943398Z (EngineCore_DP0 pid=2314) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:39.2944055Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:39.2944668Z (EngineCore_DP0 pid=2314) self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:39.2945235Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2945897Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:39.2946482Z (EngineCore_DP0 pid=2314) self._init_executor() 2025-10-10T01:39:39.2947145Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:39.2947827Z (EngineCore_DP0 pid=2314) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:39.2948553Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:39.2949265Z (EngineCore_DP0 pid=2314) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:39.2949760Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2950399Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:39.2950990Z (EngineCore_DP0 pid=2314) return func(*args, **kwargs) 2025-10-10T01:39:39.2951364Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2951998Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:39.2952693Z (EngineCore_DP0 pid=2314) worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:39.2953116Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2953788Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:39.2954428Z (EngineCore_DP0 pid=2314) module = importlib.import_module(module_name) 2025-10-10T01:39:39.2954882Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2955465Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:39.2956088Z (EngineCore_DP0 pid=2314) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:39.2956579Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2957067Z (EngineCore_DP0 pid=2314) File "", line 1387, in _gcd_import 2025-10-10T01:39:39.2957675Z (EngineCore_DP0 pid=2314) File "", line 1360, in _find_and_load 2025-10-10T01:39:39.2958264Z (EngineCore_DP0 pid=2314) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:39.2958842Z (EngineCore_DP0 pid=2314) File "", line 935, in _load_unlocked 2025-10-10T01:39:39.2959500Z (EngineCore_DP0 pid=2314) File "", line 999, in exec_module 2025-10-10T01:39:39.2960092Z (EngineCore_DP0 pid=2314) File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:39.2960824Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:39.2961500Z (EngineCore_DP0 pid=2314) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:39.2962303Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:39.2963026Z (EngineCore_DP0 pid=2314) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:39.2963805Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:39.2964479Z (EngineCore_DP0 pid=2314) class FlashAttentionMetadataBuilder( 2025-10-10T01:39:39.2965271Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:39.2966088Z (EngineCore_DP0 pid=2314) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:39.2966571Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2967264Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:39.2967941Z (EngineCore_DP0 pid=2314) if not is_fa_version_supported(fa_version): 2025-10-10T01:39:39.2968379Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2969120Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:39.2969858Z (EngineCore_DP0 pid=2314) return _is_fa2_supported(device)[0] 2025-10-10T01:39:39.2970285Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2970988Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:39.2971710Z (EngineCore_DP0 pid=2314) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:39.2972165Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2972834Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:39.2973468Z (EngineCore_DP0 pid=2314) prop = get_device_properties(device) 2025-10-10T01:39:39.2973879Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:39.2974538Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:39.2975198Z (EngineCore_DP0 pid=2314) _lazy_init() # will define _get_device_properties 2025-10-10T01:39:39.2975642Z (EngineCore_DP0 pid=2314) ^^^^^^^^^^^^ 2025-10-10T01:39:39.2976230Z (EngineCore_DP0 pid=2314) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:39.2976789Z (EngineCore_DP0 pid=2314) raise RuntimeError( 2025-10-10T01:39:39.2977468Z (EngineCore_DP0 pid=2314) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:39.6945823Z FAILED 2025-10-10T01:39:39.7076719Z models/test_initialization.py::test_can_initialize_large_subset[ErnieMTPModel] Fork a new process to run a test 2318 2025-10-10T01:39:39.7086687Z Fork a new process to run a test 0 2025-10-10T01:39:39.7359805Z INFO 10-10 01:39:39 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ErnieMTPModel', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'baidu/ERNIE-4.5-21B-A3B-PT', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'baidu/ERNIE-4.5-21B-A3B-PT'} 2025-10-10T01:39:39.8229414Z 2025-10-10T01:39:39.8231177Z config.json: 0% 0.00/936 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:39:55.3167825Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] EngineCore failed to start. 2025-10-10T01:39:55.3168456Z 2025-10-10T01:39:55.3169081Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] Traceback (most recent call last): 2025-10-10T01:39:55.3169699Z 2025-10-10T01:39:55.3170399Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:55.3170988Z 2025-10-10T01:39:55.3171371Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:55.3171971Z 2025-10-10T01:39:55.3172329Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3172677Z 2025-10-10T01:39:55.3173283Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:55.3173837Z 2025-10-10T01:39:55.3174256Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:55.3174863Z 2025-10-10T01:39:55.3175718Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:55.3176179Z 2025-10-10T01:39:55.3176518Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:55.3176826Z 2025-10-10T01:39:55.3177107Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3177498Z 2025-10-10T01:39:55.3178159Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:55.3178997Z 2025-10-10T01:39:55.3179374Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] self._init_executor() 2025-10-10T01:39:55.3179845Z 2025-10-10T01:39:55.3180816Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:55.3181693Z 2025-10-10T01:39:55.3182267Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:55.3182719Z 2025-10-10T01:39:55.3183356Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:55.3183892Z 2025-10-10T01:39:55.3184415Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:55.3184765Z 2025-10-10T01:39:55.3185057Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3185341Z 2025-10-10T01:39:55.3185827Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:55.3186304Z 2025-10-10T01:39:55.3186583Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:39:55.3186858Z 2025-10-10T01:39:55.3187116Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3187378Z 2025-10-10T01:39:55.3187918Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:55.3188408Z 2025-10-10T01:39:55.3188697Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:55.3188997Z 2025-10-10T01:39:55.3189263Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3189538Z 2025-10-10T01:39:55.3190063Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:55.3190598Z 2025-10-10T01:39:55.3190901Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:39:55.3191201Z 2025-10-10T01:39:55.3191480Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3191771Z 2025-10-10T01:39:55.3192226Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:55.3192646Z 2025-10-10T01:39:55.3193032Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:55.3193365Z 2025-10-10T01:39:55.3193650Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3193935Z 2025-10-10T01:39:55.3194273Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:39:55.3194669Z 2025-10-10T01:39:55.3195026Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:39:55.3195375Z 2025-10-10T01:39:55.3195746Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:55.3196293Z 2025-10-10T01:39:55.3196639Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:39:55.3196980Z 2025-10-10T01:39:55.3197343Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:39:55.3197695Z 2025-10-10T01:39:55.3198074Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:55.3198437Z 2025-10-10T01:39:55.3199448Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:55.3199958Z 2025-10-10T01:39:55.3200354Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:55.3200696Z 2025-10-10T01:39:55.3201225Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:55.3201705Z 2025-10-10T01:39:55.3202076Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:55.3202440Z 2025-10-10T01:39:55.3202997Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:55.3203527Z 2025-10-10T01:39:55.3203820Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:39:55.3204129Z 2025-10-10T01:39:55.3204761Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:55.3205338Z 2025-10-10T01:39:55.3205705Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:55.3206141Z 2025-10-10T01:39:55.3206400Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3206665Z 2025-10-10T01:39:55.3207237Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:55.3207750Z 2025-10-10T01:39:55.3208057Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:39:55.3208350Z 2025-10-10T01:39:55.3208627Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3208895Z 2025-10-10T01:39:55.3209496Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:55.3210049Z 2025-10-10T01:39:55.3210337Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:39:55.3210709Z 2025-10-10T01:39:55.3210976Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3211252Z 2025-10-10T01:39:55.3211843Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:55.3212379Z 2025-10-10T01:39:55.3212696Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:55.3213009Z 2025-10-10T01:39:55.3213291Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3213564Z 2025-10-10T01:39:55.3214108Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:55.3214595Z 2025-10-10T01:39:55.3214954Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:39:55.3215253Z 2025-10-10T01:39:55.3215519Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3215795Z 2025-10-10T01:39:55.3216318Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:55.3216805Z 2025-10-10T01:39:55.3217111Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:39:55.3217418Z 2025-10-10T01:39:55.3217639Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:39:55.3217880Z 2025-10-10T01:39:55.3218363Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:55.3218810Z 2025-10-10T01:39:55.3219058Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] raise RuntimeError( 2025-10-10T01:39:55.3219312Z 2025-10-10T01:39:55.3219879Z (EngineCore_DP0 pid=2447) ERROR 10-10 01:39:55 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:55.3220563Z (EngineCore_DP0 pid=2447) Process EngineCore_DP0: 2025-10-10T01:39:55.3221015Z (EngineCore_DP0 pid=2447) Traceback (most recent call last): 2025-10-10T01:39:55.3221625Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:39:55.3222142Z (EngineCore_DP0 pid=2447) self.run() 2025-10-10T01:39:55.3222670Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:39:55.3223225Z (EngineCore_DP0 pid=2447) self._target(*self._args, **self._kwargs) 2025-10-10T01:39:55.3223889Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:39:55.3224455Z (EngineCore_DP0 pid=2447) raise e 2025-10-10T01:39:55.3225056Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:39:55.3225697Z (EngineCore_DP0 pid=2447) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:39:55.3226132Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3226807Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:39:55.3227454Z (EngineCore_DP0 pid=2447) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:39:55.3228128Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:39:55.3228757Z (EngineCore_DP0 pid=2447) self.model_executor = executor_class(vllm_config) 2025-10-10T01:39:55.3229215Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3229859Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:39:55.3230446Z (EngineCore_DP0 pid=2447) self._init_executor() 2025-10-10T01:39:55.3231181Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:39:55.3231873Z (EngineCore_DP0 pid=2447) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:39:55.3232598Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:39:55.3233301Z (EngineCore_DP0 pid=2447) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:39:55.3233797Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3234444Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:39:55.3235029Z (EngineCore_DP0 pid=2447) return func(*args, **kwargs) 2025-10-10T01:39:55.3235428Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3236060Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:39:55.3236688Z (EngineCore_DP0 pid=2447) worker_class = resolve_obj_by_qualname( 2025-10-10T01:39:55.3237112Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3237775Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:39:55.3238478Z (EngineCore_DP0 pid=2447) module = importlib.import_module(module_name) 2025-10-10T01:39:55.3238915Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3239605Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:39:55.3240236Z (EngineCore_DP0 pid=2447) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:39:55.3240729Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3241224Z (EngineCore_DP0 pid=2447) File "", line 1387, in _gcd_import 2025-10-10T01:39:55.3241772Z (EngineCore_DP0 pid=2447) File "", line 1360, in _find_and_load 2025-10-10T01:39:55.3242357Z (EngineCore_DP0 pid=2447) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:39:55.3242948Z (EngineCore_DP0 pid=2447) File "", line 935, in _load_unlocked 2025-10-10T01:39:55.3243514Z (EngineCore_DP0 pid=2447) File "", line 999, in exec_module 2025-10-10T01:39:55.3244171Z (EngineCore_DP0 pid=2447) File "", line 488, in _call_with_frames_removed 2025-10-10T01:39:55.3244898Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:39:55.3245573Z (EngineCore_DP0 pid=2447) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:39:55.3246291Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:39:55.3247010Z (EngineCore_DP0 pid=2447) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:39:55.3247788Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:39:55.3248457Z (EngineCore_DP0 pid=2447) class FlashAttentionMetadataBuilder( 2025-10-10T01:39:55.3249306Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:39:55.3250122Z (EngineCore_DP0 pid=2447) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:39:55.3250608Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3251296Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:39:55.3251978Z (EngineCore_DP0 pid=2447) if not is_fa_version_supported(fa_version): 2025-10-10T01:39:55.3252410Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3253151Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:39:55.3253861Z (EngineCore_DP0 pid=2447) return _is_fa2_supported(device)[0] 2025-10-10T01:39:55.3254272Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3254980Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:39:55.3255694Z (EngineCore_DP0 pid=2447) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:39:55.3256188Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3256871Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:39:55.3257515Z (EngineCore_DP0 pid=2447) prop = get_device_properties(device) 2025-10-10T01:39:55.3257935Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:39:55.3258595Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:39:55.3259261Z (EngineCore_DP0 pid=2447) _lazy_init() # will define _get_device_properties 2025-10-10T01:39:55.3259662Z (EngineCore_DP0 pid=2447) ^^^^^^^^^^^^ 2025-10-10T01:39:55.3260248Z (EngineCore_DP0 pid=2447) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:39:55.3260810Z (EngineCore_DP0 pid=2447) raise RuntimeError( 2025-10-10T01:39:55.3261493Z (EngineCore_DP0 pid=2447) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:39:55.7232691Z FAILED 2025-10-10T01:39:55.7362641Z models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEForCausalLM] Fork a new process to run a test 2451 2025-10-10T01:39:55.7373460Z Fork a new process to run a test 0 2025-10-10T01:39:55.7377710Z `transformers==4.56.2` installed, but `transformers>=4.57.0.dev0` is required to run this model. 2025-10-10T01:39:56.0377789Z PASSED 2025-10-10T01:39:56.0506192Z models/test_initialization.py::test_can_initialize_large_subset[Glm4MoeMTPModel] Fork a new process to run a test 2452 2025-10-10T01:39:56.0516141Z Fork a new process to run a test 0 2025-10-10T01:39:56.0517797Z Model is not available online 2025-10-10T01:39:56.3564622Z PASSED 2025-10-10T01:39:56.3693948Z models/test_initialization.py::test_can_initialize_large_subset[GptOssForCausalLM] Fork a new process to run a test 2453 2025-10-10T01:39:56.3704844Z Fork a new process to run a test 0 2025-10-10T01:39:56.3981750Z INFO 10-10 01:39:56 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GptOssForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'lmsys/gpt-oss-20b-bf16'} 2025-10-10T01:39:56.5016198Z 2025-10-10T01:39:56.5018361Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:39:56.5018694Z config.json: 1.59kB [00:00, 9.04MB/s] 2025-10-10T01:40:03.1706919Z INFO 10-10 01:40:03 [model.py:551] Resolved architecture: GptOssForCausalLM 2025-10-10T01:40:03.1707669Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:40:03.1951463Z INFO 10-10 01:40:03 [model.py:1545] Using max model len 131072 2025-10-10T01:40:03.3817835Z INFO 10-10 01:40:03 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:40:03.3818692Z INFO 10-10 01:40:03 [config.py:271] Overriding max cuda graph capture size to 992 for performance. 2025-10-10T01:40:03.4300486Z 2025-10-10T01:40:03.4302427Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:03.4302807Z tokenizer_config.json: 4.20kB [00:00, 26.3MB/s] 2025-10-10T01:40:03.8234670Z 2025-10-10T01:40:04.0454063Z tokenizer.json: 0% 0.00/27.9M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [992, 976, 960, 944, 928, 912, 896, 880, 864, 848, 832, 816, 800, 784, 768, 752, 736, 720, 704, 688, 672, 656, 640, 624, 608, 592, 576, 560, 544, 528, 512, 496, 480, 464, 448, 432, 416, 400, 384, 368, 352, 336, 320, 304, 288, 272, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 992, 'local_cache_dir': None} 2025-10-10T01:40:05.2140408Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] EngineCore failed to start. 2025-10-10T01:40:05.2140964Z 2025-10-10T01:40:05.2141511Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] Traceback (most recent call last): 2025-10-10T01:40:05.2141900Z 2025-10-10T01:40:05.2142757Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:05.2143350Z 2025-10-10T01:40:05.2143688Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:05.2143996Z 2025-10-10T01:40:05.2144270Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2144559Z 2025-10-10T01:40:05.2145040Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:05.2145506Z 2025-10-10T01:40:05.2145836Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:05.2146340Z 2025-10-10T01:40:05.2146834Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:05.2147279Z 2025-10-10T01:40:05.2147706Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:05.2148042Z 2025-10-10T01:40:05.2148335Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2148752Z 2025-10-10T01:40:05.2149274Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:05.2149741Z 2025-10-10T01:40:05.2149992Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] self._init_executor() 2025-10-10T01:40:05.2150262Z 2025-10-10T01:40:05.2150799Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:05.2151455Z 2025-10-10T01:40:05.2151991Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:05.2152394Z 2025-10-10T01:40:05.2153228Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:05.2153918Z 2025-10-10T01:40:05.2154297Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:05.2154773Z 2025-10-10T01:40:05.2155164Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2155603Z 2025-10-10T01:40:05.2156318Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:05.2156788Z 2025-10-10T01:40:05.2157268Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:40:05.2157561Z 2025-10-10T01:40:05.2157831Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2158107Z 2025-10-10T01:40:05.2158640Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:05.2159262Z 2025-10-10T01:40:05.2159553Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:05.2159845Z 2025-10-10T01:40:05.2160110Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2160387Z 2025-10-10T01:40:05.2160922Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:05.2161404Z 2025-10-10T01:40:05.2161709Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:40:05.2162008Z 2025-10-10T01:40:05.2162289Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2162556Z 2025-10-10T01:40:05.2162999Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:05.2163472Z 2025-10-10T01:40:05.2163812Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:05.2164145Z 2025-10-10T01:40:05.2164449Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2172366Z 2025-10-10T01:40:05.2172822Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:40:05.2173198Z 2025-10-10T01:40:05.2173580Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:40:05.2173925Z 2025-10-10T01:40:05.2174325Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:05.2174705Z 2025-10-10T01:40:05.2175068Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:40:05.2175504Z 2025-10-10T01:40:05.2175880Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:40:05.2176240Z 2025-10-10T01:40:05.2176613Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:05.2176979Z 2025-10-10T01:40:05.2177486Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:05.2177954Z 2025-10-10T01:40:05.2178304Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:05.2178653Z 2025-10-10T01:40:05.2179192Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:05.2179673Z 2025-10-10T01:40:05.2180139Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:05.2180504Z 2025-10-10T01:40:05.2181082Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:05.2181598Z 2025-10-10T01:40:05.2181914Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:40:05.2182210Z 2025-10-10T01:40:05.2182858Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:05.2183465Z 2025-10-10T01:40:05.2183839Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:05.2184199Z 2025-10-10T01:40:05.2184453Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2184727Z 2025-10-10T01:40:05.2185292Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:05.2185814Z 2025-10-10T01:40:05.2186112Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:40:05.2186466Z 2025-10-10T01:40:05.2186740Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2187011Z 2025-10-10T01:40:05.2187623Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:05.2188190Z 2025-10-10T01:40:05.2188478Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:40:05.2188757Z 2025-10-10T01:40:05.2189023Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2189283Z 2025-10-10T01:40:05.2189857Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:05.2190389Z 2025-10-10T01:40:05.2190706Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:05.2191079Z 2025-10-10T01:40:05.2191357Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2191635Z 2025-10-10T01:40:05.2192156Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:05.2192641Z 2025-10-10T01:40:05.2192925Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:40:05.2193233Z 2025-10-10T01:40:05.2193493Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2193769Z 2025-10-10T01:40:05.2194286Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:05.2194761Z 2025-10-10T01:40:05.2195157Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:05.2195464Z 2025-10-10T01:40:05.2195711Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:40:05.2195955Z 2025-10-10T01:40:05.2196894Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:05.2197357Z 2025-10-10T01:40:05.2197620Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] raise RuntimeError( 2025-10-10T01:40:05.2197892Z 2025-10-10T01:40:05.2198459Z (EngineCore_DP0 pid=2531) ERROR 10-10 01:40:05 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:05.2199268Z (EngineCore_DP0 pid=2531) Process EngineCore_DP0: 2025-10-10T01:40:05.2199684Z (EngineCore_DP0 pid=2531) Traceback (most recent call last): 2025-10-10T01:40:05.2200322Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:40:05.2200848Z (EngineCore_DP0 pid=2531) self.run() 2025-10-10T01:40:05.2201371Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:40:05.2201939Z (EngineCore_DP0 pid=2531) self._target(*self._args, **self._kwargs) 2025-10-10T01:40:05.2202759Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:40:05.2203321Z (EngineCore_DP0 pid=2531) raise e 2025-10-10T01:40:05.2203923Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:05.2204562Z (EngineCore_DP0 pid=2531) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:05.2205008Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2205643Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:05.2206285Z (EngineCore_DP0 pid=2531) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:05.2206946Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:05.2207557Z (EngineCore_DP0 pid=2531) self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:05.2208105Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2208764Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:05.2209355Z (EngineCore_DP0 pid=2531) self._init_executor() 2025-10-10T01:40:05.2210017Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:05.2210695Z (EngineCore_DP0 pid=2531) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:05.2211425Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:05.2212144Z (EngineCore_DP0 pid=2531) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:05.2212653Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2213420Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:05.2214026Z (EngineCore_DP0 pid=2531) return func(*args, **kwargs) 2025-10-10T01:40:05.2214408Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2215044Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:05.2215674Z (EngineCore_DP0 pid=2531) worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:05.2216099Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2216764Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:05.2217419Z (EngineCore_DP0 pid=2531) module = importlib.import_module(module_name) 2025-10-10T01:40:05.2217860Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2218437Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:05.2219060Z (EngineCore_DP0 pid=2531) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:05.2219555Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2220103Z (EngineCore_DP0 pid=2531) File "", line 1387, in _gcd_import 2025-10-10T01:40:05.2220661Z (EngineCore_DP0 pid=2531) File "", line 1360, in _find_and_load 2025-10-10T01:40:05.2221255Z (EngineCore_DP0 pid=2531) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:05.2221836Z (EngineCore_DP0 pid=2531) File "", line 935, in _load_unlocked 2025-10-10T01:40:05.2222406Z (EngineCore_DP0 pid=2531) File "", line 999, in exec_module 2025-10-10T01:40:05.2222996Z (EngineCore_DP0 pid=2531) File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:05.2223723Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:05.2224403Z (EngineCore_DP0 pid=2531) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:05.2225138Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:05.2226546Z (EngineCore_DP0 pid=2531) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:05.2227321Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:05.2227993Z (EngineCore_DP0 pid=2531) class FlashAttentionMetadataBuilder( 2025-10-10T01:40:05.2228782Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:05.2229600Z (EngineCore_DP0 pid=2531) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:05.2230090Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2230781Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:05.2231554Z (EngineCore_DP0 pid=2531) if not is_fa_version_supported(fa_version): 2025-10-10T01:40:05.2232021Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2232771Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:05.2233471Z (EngineCore_DP0 pid=2531) return _is_fa2_supported(device)[0] 2025-10-10T01:40:05.2233884Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2234587Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:05.2235304Z (EngineCore_DP0 pid=2531) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:05.2235768Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2236440Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:05.2237077Z (EngineCore_DP0 pid=2531) prop = get_device_properties(device) 2025-10-10T01:40:05.2237492Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:05.2238150Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:05.2238861Z (EngineCore_DP0 pid=2531) _lazy_init() # will define _get_device_properties 2025-10-10T01:40:05.2239331Z (EngineCore_DP0 pid=2531) ^^^^^^^^^^^^ 2025-10-10T01:40:05.2239929Z (EngineCore_DP0 pid=2531) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:05.2240484Z (EngineCore_DP0 pid=2531) raise RuntimeError( 2025-10-10T01:40:05.2241175Z (EngineCore_DP0 pid=2531) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:05.6488590Z FAILED 2025-10-10T01:40:05.6618059Z models/test_initialization.py::test_can_initialize_large_subset[SmolVLMForConditionalGeneration] Fork a new process to run a test 2535 2025-10-10T01:40:05.6629062Z Fork a new process to run a test 0 2025-10-10T01:40:05.6906622Z INFO 10-10 01:40:05 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='SmolVLMForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'HuggingFaceTB/SmolVLM2-2.2B-Instruct'} 2025-10-10T01:40:05.8219327Z 2025-10-10T01:40:05.8221358Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:05.8221760Z config.json: 3.64kB [00:00, 19.4MB/s] 2025-10-10T01:40:05.9356068Z 2025-10-10T01:40:05.9357175Z preprocessor_config.json: 0% 0.00/599 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:40:13.7071632Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] EngineCore failed to start. 2025-10-10T01:40:13.7072113Z 2025-10-10T01:40:13.7072509Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] Traceback (most recent call last): 2025-10-10T01:40:13.7072868Z 2025-10-10T01:40:13.7073541Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:13.7074144Z 2025-10-10T01:40:13.7074853Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:13.7075284Z 2025-10-10T01:40:13.7075658Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7076006Z 2025-10-10T01:40:13.7076621Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:13.7077190Z 2025-10-10T01:40:13.7077617Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:13.7078040Z 2025-10-10T01:40:13.7078642Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:13.7079350Z 2025-10-10T01:40:13.7079754Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:13.7080157Z 2025-10-10T01:40:13.7080506Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7080847Z 2025-10-10T01:40:13.7081703Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:13.7082764Z 2025-10-10T01:40:13.7083217Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] self._init_executor() 2025-10-10T01:40:13.7083712Z 2025-10-10T01:40:13.7084739Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:13.7085458Z 2025-10-10T01:40:13.7085812Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:13.7086141Z 2025-10-10T01:40:13.7086688Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:13.7087189Z 2025-10-10T01:40:13.7087533Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:13.7087873Z 2025-10-10T01:40:13.7088163Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7088450Z 2025-10-10T01:40:13.7088938Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:13.7089532Z 2025-10-10T01:40:13.7089819Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:40:13.7090097Z 2025-10-10T01:40:13.7090363Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7090621Z 2025-10-10T01:40:13.7091168Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:13.7091652Z 2025-10-10T01:40:13.7091942Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:13.7092237Z 2025-10-10T01:40:13.7092502Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7092779Z 2025-10-10T01:40:13.7093407Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:13.7093901Z 2025-10-10T01:40:13.7094208Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:40:13.7094514Z 2025-10-10T01:40:13.7094783Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7095054Z 2025-10-10T01:40:13.7095496Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:13.7095902Z 2025-10-10T01:40:13.7096458Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:13.7096793Z 2025-10-10T01:40:13.7097090Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7097367Z 2025-10-10T01:40:13.7097706Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:40:13.7098039Z 2025-10-10T01:40:13.7098384Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:40:13.7099436Z 2025-10-10T01:40:13.7099823Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:13.7100187Z 2025-10-10T01:40:13.7100531Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:40:13.7100870Z 2025-10-10T01:40:13.7101237Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:40:13.7101593Z 2025-10-10T01:40:13.7101970Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:13.7102324Z 2025-10-10T01:40:13.7102832Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:13.7103295Z 2025-10-10T01:40:13.7103651Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:13.7103987Z 2025-10-10T01:40:13.7104506Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:13.7105070Z 2025-10-10T01:40:13.7105446Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:13.7105819Z 2025-10-10T01:40:13.7106367Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:13.7106898Z 2025-10-10T01:40:13.7107203Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:40:13.7107504Z 2025-10-10T01:40:13.7108136Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:13.7108713Z 2025-10-10T01:40:13.7109242Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:13.7109602Z 2025-10-10T01:40:13.7109872Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7110129Z 2025-10-10T01:40:13.7110695Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:13.7111219Z 2025-10-10T01:40:13.7111518Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:40:13.7111811Z 2025-10-10T01:40:13.7112082Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7112369Z 2025-10-10T01:40:13.7112972Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:13.7113524Z 2025-10-10T01:40:13.7113813Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:40:13.7114098Z 2025-10-10T01:40:13.7114355Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7114615Z 2025-10-10T01:40:13.7115259Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:13.7115784Z 2025-10-10T01:40:13.7116106Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:13.7116417Z 2025-10-10T01:40:13.7116698Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7116979Z 2025-10-10T01:40:13.7117505Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:13.7117985Z 2025-10-10T01:40:13.7118264Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:40:13.7118557Z 2025-10-10T01:40:13.7118814Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7119199Z 2025-10-10T01:40:13.7119721Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:13.7120253Z 2025-10-10T01:40:13.7120563Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:13.7120874Z 2025-10-10T01:40:13.7121106Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:40:13.7121342Z 2025-10-10T01:40:13.7121824Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:13.7122271Z 2025-10-10T01:40:13.7122525Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] raise RuntimeError( 2025-10-10T01:40:13.7122783Z 2025-10-10T01:40:13.7123349Z (EngineCore_DP0 pid=2593) ERROR 10-10 01:40:13 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:13.7124150Z (EngineCore_DP0 pid=2593) Process EngineCore_DP0: 2025-10-10T01:40:13.7124548Z (EngineCore_DP0 pid=2593) Traceback (most recent call last): 2025-10-10T01:40:13.7125166Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:40:13.7125683Z (EngineCore_DP0 pid=2593) self.run() 2025-10-10T01:40:13.7126205Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:40:13.7126766Z (EngineCore_DP0 pid=2593) self._target(*self._args, **self._kwargs) 2025-10-10T01:40:13.7127417Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:40:13.7127972Z (EngineCore_DP0 pid=2593) raise e 2025-10-10T01:40:13.7128560Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:13.7129188Z (EngineCore_DP0 pid=2593) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:13.7129622Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7130227Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:13.7130876Z (EngineCore_DP0 pid=2593) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:13.7131583Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:13.7132192Z (EngineCore_DP0 pid=2593) self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:13.7132648Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7133297Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:13.7133879Z (EngineCore_DP0 pid=2593) self._init_executor() 2025-10-10T01:40:13.7134534Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:13.7135214Z (EngineCore_DP0 pid=2593) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:13.7135937Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:13.7136642Z (EngineCore_DP0 pid=2593) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:13.7137181Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7137820Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:13.7138415Z (EngineCore_DP0 pid=2593) return func(*args, **kwargs) 2025-10-10T01:40:13.7138808Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7139437Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:13.7140056Z (EngineCore_DP0 pid=2593) worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:13.7140484Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7141230Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:13.7141906Z (EngineCore_DP0 pid=2593) module = importlib.import_module(module_name) 2025-10-10T01:40:13.7142353Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7142931Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:13.7143555Z (EngineCore_DP0 pid=2593) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:13.7144046Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7144536Z (EngineCore_DP0 pid=2593) File "", line 1387, in _gcd_import 2025-10-10T01:40:13.7145093Z (EngineCore_DP0 pid=2593) File "", line 1360, in _find_and_load 2025-10-10T01:40:13.7145676Z (EngineCore_DP0 pid=2593) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:13.7146262Z (EngineCore_DP0 pid=2593) File "", line 935, in _load_unlocked 2025-10-10T01:40:13.7146831Z (EngineCore_DP0 pid=2593) File "", line 999, in exec_module 2025-10-10T01:40:13.7147432Z (EngineCore_DP0 pid=2593) File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:13.7148161Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:13.7148876Z (EngineCore_DP0 pid=2593) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:13.7149591Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:13.7150327Z (EngineCore_DP0 pid=2593) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:13.7151112Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:13.7151782Z (EngineCore_DP0 pid=2593) class FlashAttentionMetadataBuilder( 2025-10-10T01:40:13.7152570Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:13.7153368Z (EngineCore_DP0 pid=2593) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:13.7153847Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7154579Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:13.7155257Z (EngineCore_DP0 pid=2593) if not is_fa_version_supported(fa_version): 2025-10-10T01:40:13.7155706Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7156437Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:13.7157128Z (EngineCore_DP0 pid=2593) return _is_fa2_supported(device)[0] 2025-10-10T01:40:13.7157537Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7158239Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:13.7159092Z (EngineCore_DP0 pid=2593) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:13.7159563Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7160218Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:13.7160845Z (EngineCore_DP0 pid=2593) prop = get_device_properties(device) 2025-10-10T01:40:13.7161285Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:13.7161942Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:13.7162593Z (EngineCore_DP0 pid=2593) _lazy_init() # will define _get_device_properties 2025-10-10T01:40:13.7162982Z (EngineCore_DP0 pid=2593) ^^^^^^^^^^^^ 2025-10-10T01:40:13.7163560Z (EngineCore_DP0 pid=2593) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:13.7164132Z (EngineCore_DP0 pid=2593) raise RuntimeError( 2025-10-10T01:40:13.7164817Z (EngineCore_DP0 pid=2593) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:14.1057358Z FAILED 2025-10-10T01:40:14.1184683Z models/test_initialization.py::test_can_initialize_large_subset[LlavaNextForConditionalGeneration] Fork a new process to run a test 2597 2025-10-10T01:40:14.1196005Z Fork a new process to run a test 0 2025-10-10T01:40:14.1473692Z INFO 10-10 01:40:14 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlavaNextForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'llava-hf/llava-v1.6-mistral-7b-hf'} 2025-10-10T01:40:14.2880713Z 2025-10-10T01:40:14.2882647Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:14.2882961Z config.json: 1.28kB [00:00, 6.99MB/s] 2025-10-10T01:40:14.5985696Z 2025-10-10T01:40:14.5986510Z preprocessor_config.json: 0% 0.00/772 [00:00", line 1387, in _gcd_import 2025-10-10T01:40:22.7913684Z 2025-10-10T01:40:22.7913894Z ERROR 10-10 01:40:22 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:40:22.7914177Z 2025-10-10T01:40:22.7914409Z ERROR 10-10 01:40:22 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:22.7914708Z 2025-10-10T01:40:22.7914910Z ERROR 10-10 01:40:22 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:40:22.7915178Z 2025-10-10T01:40:22.7915398Z ERROR 10-10 01:40:22 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:40:22.7915698Z 2025-10-10T01:40:22.7915938Z ERROR 10-10 01:40:22 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:22.7916243Z 2025-10-10T01:40:22.7916615Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava_next.py", line 24, in 2025-10-10T01:40:22.7917148Z 2025-10-10T01:40:22.7917391Z ERROR 10-10 01:40:22 [registry.py:542] from .llava import (BaseLlavaMultiModalProcessor, BaseLlavaProcessingInfo, 2025-10-10T01:40:22.7917724Z 2025-10-10T01:40:22.7918068Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava.py", line 40, in 2025-10-10T01:40:22.7918482Z 2025-10-10T01:40:22.7918694Z ERROR 10-10 01:40:22 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T01:40:22.7919088Z 2025-10-10T01:40:22.7919444Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:40:22.7919864Z 2025-10-10T01:40:22.7920011Z ERROR 10-10 01:40:22 [registry.py:542] from xformers import ops as xops 2025-10-10T01:40:22.7920222Z 2025-10-10T01:40:22.7920648Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:40:22.7921237Z 2025-10-10T01:40:22.7921402Z ERROR 10-10 01:40:22 [registry.py:542] from .fmha import ( 2025-10-10T01:40:22.7921681Z 2025-10-10T01:40:22.7922145Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:40:22.7922632Z 2025-10-10T01:40:22.7922871Z ERROR 10-10 01:40:22 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:40:22.7923180Z 2025-10-10T01:40:22.7923549Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:40:22.7923976Z 2025-10-10T01:40:22.7924142Z ERROR 10-10 01:40:22 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:40:22.7924374Z 2025-10-10T01:40:22.7924524Z ERROR 10-10 01:40:22 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:22.7924733Z 2025-10-10T01:40:22.7925050Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:40:22.7925438Z 2025-10-10T01:40:22.7925545Z ERROR 10-10 01:40:22 [registry.py:542] value = func() 2025-10-10T01:40:22.7925726Z 2025-10-10T01:40:22.7925826Z ERROR 10-10 01:40:22 [registry.py:542] ^^^^^^ 2025-10-10T01:40:22.7926057Z 2025-10-10T01:40:22.7926400Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:40:22.7926808Z 2025-10-10T01:40:22.7926996Z ERROR 10-10 01:40:22 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:40:22.7927258Z 2025-10-10T01:40:22.7927403Z ERROR 10-10 01:40:22 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:22.7927629Z 2025-10-10T01:40:22.7927988Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:22.7928421Z 2025-10-10T01:40:22.7928569Z ERROR 10-10 01:40:22 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:40:22.7928801Z 2025-10-10T01:40:22.7928934Z ERROR 10-10 01:40:22 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:22.7929145Z 2025-10-10T01:40:22.7929489Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:22.7929923Z 2025-10-10T01:40:22.7930094Z ERROR 10-10 01:40:22 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:22.7930392Z 2025-10-10T01:40:22.7930491Z ERROR 10-10 01:40:22 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:40:22.7930672Z 2025-10-10T01:40:22.7930987Z ERROR 10-10 01:40:22 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:22.7931371Z 2025-10-10T01:40:22.7931494Z ERROR 10-10 01:40:22 [registry.py:542] raise RuntimeError( 2025-10-10T01:40:22.7931682Z 2025-10-10T01:40:22.7932080Z ERROR 10-10 01:40:22 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:23.1638562Z FAILED 2025-10-10T01:40:23.1765602Z models/test_initialization.py::test_can_initialize_large_subset[Step3TextForCausalLM] Fork a new process to run a test 2673 2025-10-10T01:40:23.1776674Z Fork a new process to run a test 0 2025-10-10T01:40:23.2046277Z INFO 10-10 01:40:23 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Step3TextForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'stepfun-ai/step3'} 2025-10-10T01:40:23.3633867Z 2025-10-10T01:40:23.3636352Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:23.3636657Z config.json: 1.54kB [00:00, 7.43MB/s] 2025-10-10T01:40:30.1740322Z INFO 10-10 01:40:30 [model.py:551] Resolved architecture: Step3VLForConditionalGeneration 2025-10-10T01:40:30.1740758Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:40:30.1978074Z INFO 10-10 01:40:30 [model.py:1545] Using max model len 65536 2025-10-10T01:40:30.1980578Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:40:30.3653716Z INFO 10-10 01:40:30 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:40:30.4028967Z 2025-10-10T01:40:30.4030670Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:30.4031093Z tokenizer_config.json: 3.94kB [00:00, 27.5MB/s] 2025-10-10T01:40:30.5567827Z 2025-10-10T01:40:30.5829104Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:30.5829622Z tokenizer.json: 7.85MB [00:00, 300MB/s] 2025-10-10T01:40:31.0961243Z 2025-10-10T01:40:31.0961666Z generation_config.json: 0% 0.00/158 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:40:31.7009024Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] EngineCore failed to start. 2025-10-10T01:40:31.7009583Z 2025-10-10T01:40:31.7010086Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] Traceback (most recent call last): 2025-10-10T01:40:31.7010837Z 2025-10-10T01:40:31.7011739Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:31.7012496Z 2025-10-10T01:40:31.7013058Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:31.7013450Z 2025-10-10T01:40:31.7013954Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7014367Z 2025-10-10T01:40:31.7015147Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:31.7015869Z 2025-10-10T01:40:31.7016327Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:31.7016869Z 2025-10-10T01:40:31.7017548Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:31.7018008Z 2025-10-10T01:40:31.7018465Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:31.7018781Z 2025-10-10T01:40:31.7019197Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7019659Z 2025-10-10T01:40:31.7020351Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:31.7020945Z 2025-10-10T01:40:31.7021241Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] self._init_executor() 2025-10-10T01:40:31.7021509Z 2025-10-10T01:40:31.7022182Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:31.7022804Z 2025-10-10T01:40:31.7023146Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:31.7023593Z 2025-10-10T01:40:31.7024178Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:31.7024797Z 2025-10-10T01:40:31.7025267Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:31.7025741Z 2025-10-10T01:40:31.7026200Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7026489Z 2025-10-10T01:40:31.7027116Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:31.7027587Z 2025-10-10T01:40:31.7027992Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:40:31.7028277Z 2025-10-10T01:40:31.7028597Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7028938Z 2025-10-10T01:40:31.7029574Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:31.7030100Z 2025-10-10T01:40:31.7030605Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:31.7030928Z 2025-10-10T01:40:31.7031323Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7031627Z 2025-10-10T01:40:31.7032277Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:31.7032778Z 2025-10-10T01:40:31.7033227Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:40:31.7033531Z 2025-10-10T01:40:31.7033926Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7034224Z 2025-10-10T01:40:31.7034767Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:31.7035206Z 2025-10-10T01:40:31.7035668Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:31.7036009Z 2025-10-10T01:40:31.7036321Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7036716Z 2025-10-10T01:40:31.7037080Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:40:31.7037618Z 2025-10-10T01:40:31.7037980Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:40:31.7038459Z 2025-10-10T01:40:31.7038857Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:31.7039454Z 2025-10-10T01:40:31.7039833Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:40:31.7040299Z 2025-10-10T01:40:31.7040682Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:40:31.7041161Z 2025-10-10T01:40:31.7041560Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:31.7042053Z 2025-10-10T01:40:31.7042660Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:31.7043245Z 2025-10-10T01:40:31.7043754Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:31.7044130Z 2025-10-10T01:40:31.7044789Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:31.7045394Z 2025-10-10T01:40:31.7045807Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:31.7046321Z 2025-10-10T01:40:31.7047015Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:31.7047547Z 2025-10-10T01:40:31.7047981Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:40:31.7048286Z 2025-10-10T01:40:31.7049174Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:31.7049888Z 2025-10-10T01:40:31.7050315Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:31.7050762Z 2025-10-10T01:40:31.7051034Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7051311Z 2025-10-10T01:40:31.7052044Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:31.7052580Z 2025-10-10T01:40:31.7052961Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:40:31.7053308Z 2025-10-10T01:40:31.7053589Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7053863Z 2025-10-10T01:40:31.7054459Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:31.7055010Z 2025-10-10T01:40:31.7055507Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:40:31.7055815Z 2025-10-10T01:40:31.7056091Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7056356Z 2025-10-10T01:40:31.7056958Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:31.7057508Z 2025-10-10T01:40:31.7057829Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:31.7058137Z 2025-10-10T01:40:31.7058414Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7058695Z 2025-10-10T01:40:31.7059361Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:31.7059864Z 2025-10-10T01:40:31.7060156Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:40:31.7060518Z 2025-10-10T01:40:31.7060785Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7061056Z 2025-10-10T01:40:31.7061579Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:31.7062068Z 2025-10-10T01:40:31.7062374Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:31.7062671Z 2025-10-10T01:40:31.7062902Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:40:31.7063145Z 2025-10-10T01:40:31.7063630Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:31.7064075Z 2025-10-10T01:40:31.7064320Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] raise RuntimeError( 2025-10-10T01:40:31.7064659Z 2025-10-10T01:40:31.7065230Z (EngineCore_DP0 pid=2731) ERROR 10-10 01:40:31 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:31.7065921Z (EngineCore_DP0 pid=2731) Process EngineCore_DP0: 2025-10-10T01:40:31.7066493Z (EngineCore_DP0 pid=2731) Traceback (most recent call last): 2025-10-10T01:40:31.7067128Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:40:31.7067661Z (EngineCore_DP0 pid=2731) self.run() 2025-10-10T01:40:31.7068184Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:40:31.7068757Z (EngineCore_DP0 pid=2731) self._target(*self._args, **self._kwargs) 2025-10-10T01:40:31.7069435Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:40:31.7069993Z (EngineCore_DP0 pid=2731) raise e 2025-10-10T01:40:31.7070599Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:31.7071240Z (EngineCore_DP0 pid=2731) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:31.7071744Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7072364Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:31.7073002Z (EngineCore_DP0 pid=2731) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:31.7073663Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:31.7074279Z (EngineCore_DP0 pid=2731) self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:31.7074733Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7075390Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:31.7075973Z (EngineCore_DP0 pid=2731) self._init_executor() 2025-10-10T01:40:31.7076633Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:31.7077317Z (EngineCore_DP0 pid=2731) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:31.7078100Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:31.7078810Z (EngineCore_DP0 pid=2731) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:31.7079391Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7080219Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:31.7080835Z (EngineCore_DP0 pid=2731) return func(*args, **kwargs) 2025-10-10T01:40:31.7081227Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7081857Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:31.7083059Z (EngineCore_DP0 pid=2731) worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:31.7083511Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7084178Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:31.7084832Z (EngineCore_DP0 pid=2731) module = importlib.import_module(module_name) 2025-10-10T01:40:31.7085273Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7085855Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:31.7086479Z (EngineCore_DP0 pid=2731) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:31.7086968Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7087482Z (EngineCore_DP0 pid=2731) File "", line 1387, in _gcd_import 2025-10-10T01:40:31.7088031Z (EngineCore_DP0 pid=2731) File "", line 1360, in _find_and_load 2025-10-10T01:40:31.7088615Z (EngineCore_DP0 pid=2731) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:31.7089192Z (EngineCore_DP0 pid=2731) File "", line 935, in _load_unlocked 2025-10-10T01:40:31.7089817Z (EngineCore_DP0 pid=2731) File "", line 999, in exec_module 2025-10-10T01:40:31.7090410Z (EngineCore_DP0 pid=2731) File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:31.7091137Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:31.7091811Z (EngineCore_DP0 pid=2731) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:31.7092540Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:31.7093263Z (EngineCore_DP0 pid=2731) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:31.7094045Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:31.7094719Z (EngineCore_DP0 pid=2731) class FlashAttentionMetadataBuilder( 2025-10-10T01:40:31.7095511Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:31.7096571Z (EngineCore_DP0 pid=2731) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:31.7097054Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7097740Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:31.7098434Z (EngineCore_DP0 pid=2731) if not is_fa_version_supported(fa_version): 2025-10-10T01:40:31.7098870Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7099603Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:31.7100297Z (EngineCore_DP0 pid=2731) return _is_fa2_supported(device)[0] 2025-10-10T01:40:31.7100711Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7101562Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:31.7102291Z (EngineCore_DP0 pid=2731) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:31.7102738Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7103410Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:31.7104068Z (EngineCore_DP0 pid=2731) prop = get_device_properties(device) 2025-10-10T01:40:31.7104488Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:31.7105164Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:31.7105820Z (EngineCore_DP0 pid=2731) _lazy_init() # will define _get_device_properties 2025-10-10T01:40:31.7106406Z (EngineCore_DP0 pid=2731) ^^^^^^^^^^^^ 2025-10-10T01:40:31.7107000Z (EngineCore_DP0 pid=2731) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:31.7107558Z (EngineCore_DP0 pid=2731) raise RuntimeError( 2025-10-10T01:40:31.7108244Z (EngineCore_DP0 pid=2731) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:32.1065512Z FAILED 2025-10-10T01:40:32.1192195Z models/test_initialization.py::test_can_initialize_large_subset[BaichuanForCausalLM] Fork a new process to run a test 2735 2025-10-10T01:40:32.1204054Z Fork a new process to run a test 0 2025-10-10T01:40:32.1481797Z INFO 10-10 01:40:32 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BaichuanForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'baichuan-inc/Baichuan2-7B-chat'} 2025-10-10T01:40:32.2991219Z 2025-10-10T01:40:32.2992473Z config.json: 0% 0.00/758 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:40:40.9478520Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] EngineCore failed to start. 2025-10-10T01:40:40.9478933Z 2025-10-10T01:40:40.9479452Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] Traceback (most recent call last): 2025-10-10T01:40:40.9479808Z 2025-10-10T01:40:40.9480874Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:40.9481491Z 2025-10-10T01:40:40.9481887Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:40.9482269Z 2025-10-10T01:40:40.9482633Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9483005Z 2025-10-10T01:40:40.9483624Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:40.9484289Z 2025-10-10T01:40:40.9484813Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:40.9485220Z 2025-10-10T01:40:40.9485849Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:40.9486297Z 2025-10-10T01:40:40.9486617Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:40.9486919Z 2025-10-10T01:40:40.9487220Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9487504Z 2025-10-10T01:40:40.9488184Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:40.9489242Z 2025-10-10T01:40:40.9489625Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] self._init_executor() 2025-10-10T01:40:40.9490117Z 2025-10-10T01:40:40.9491046Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:40.9491775Z 2025-10-10T01:40:40.9492141Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:40.9492476Z 2025-10-10T01:40:40.9493027Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:40.9493536Z 2025-10-10T01:40:40.9493882Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:40.9494216Z 2025-10-10T01:40:40.9494509Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9494914Z 2025-10-10T01:40:40.9495417Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:40.9495891Z 2025-10-10T01:40:40.9496401Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:40:40.9496675Z 2025-10-10T01:40:40.9496927Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9497206Z 2025-10-10T01:40:40.9497720Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:40.9498208Z 2025-10-10T01:40:40.9498545Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:40.9498835Z 2025-10-10T01:40:40.9499247Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9499526Z 2025-10-10T01:40:40.9500055Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:40.9500543Z 2025-10-10T01:40:40.9500845Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:40:40.9501147Z 2025-10-10T01:40:40.9501419Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9501696Z 2025-10-10T01:40:40.9502138Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:40.9502570Z 2025-10-10T01:40:40.9502914Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:40.9503241Z 2025-10-10T01:40:40.9503529Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9503805Z 2025-10-10T01:40:40.9504149Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:40:40.9504478Z 2025-10-10T01:40:40.9504908Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:40:40.9505243Z 2025-10-10T01:40:40.9505625Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:40.9505991Z 2025-10-10T01:40:40.9506339Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:40:40.9506692Z 2025-10-10T01:40:40.9507048Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:40:40.9507401Z 2025-10-10T01:40:40.9507770Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:40.9508131Z 2025-10-10T01:40:40.9508629Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:40.9509087Z 2025-10-10T01:40:40.9509438Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:40.9509841Z 2025-10-10T01:40:40.9510373Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:40.9510849Z 2025-10-10T01:40:40.9511222Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:40.9511575Z 2025-10-10T01:40:40.9512130Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:40.9512642Z 2025-10-10T01:40:40.9512956Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:40:40.9513269Z 2025-10-10T01:40:40.9513999Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:40.9514595Z 2025-10-10T01:40:40.9514963Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:40.9515317Z 2025-10-10T01:40:40.9515569Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9515835Z 2025-10-10T01:40:40.9516401Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:40.9516920Z 2025-10-10T01:40:40.9517215Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:40:40.9517512Z 2025-10-10T01:40:40.9517800Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9518074Z 2025-10-10T01:40:40.9518675Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:40.9519348Z 2025-10-10T01:40:40.9519631Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:40:40.9519919Z 2025-10-10T01:40:40.9520176Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9520497Z 2025-10-10T01:40:40.9521083Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:40.9521627Z 2025-10-10T01:40:40.9521948Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:40.9522267Z 2025-10-10T01:40:40.9522537Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9522810Z 2025-10-10T01:40:40.9523334Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:40.9523815Z 2025-10-10T01:40:40.9524097Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:40:40.9524378Z 2025-10-10T01:40:40.9524641Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9524955Z 2025-10-10T01:40:40.9525481Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:40.9525977Z 2025-10-10T01:40:40.9526291Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:40.9526601Z 2025-10-10T01:40:40.9526827Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:40:40.9527072Z 2025-10-10T01:40:40.9527549Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:40.9528004Z 2025-10-10T01:40:40.9528246Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] raise RuntimeError( 2025-10-10T01:40:40.9528502Z 2025-10-10T01:40:40.9529151Z (EngineCore_DP0 pid=2813) ERROR 10-10 01:40:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:40.9529852Z (EngineCore_DP0 pid=2813) Process EngineCore_DP0: 2025-10-10T01:40:40.9530238Z (EngineCore_DP0 pid=2813) Traceback (most recent call last): 2025-10-10T01:40:40.9530841Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:40:40.9531362Z (EngineCore_DP0 pid=2813) self.run() 2025-10-10T01:40:40.9531904Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:40:40.9532473Z (EngineCore_DP0 pid=2813) self._target(*self._args, **self._kwargs) 2025-10-10T01:40:40.9533131Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:40:40.9533682Z (EngineCore_DP0 pid=2813) raise e 2025-10-10T01:40:40.9534271Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:40.9534904Z (EngineCore_DP0 pid=2813) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:40.9535358Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9535973Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:40.9536688Z (EngineCore_DP0 pid=2813) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:40.9537350Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:40.9537969Z (EngineCore_DP0 pid=2813) self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:40.9538420Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9539063Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:40.9539641Z (EngineCore_DP0 pid=2813) self._init_executor() 2025-10-10T01:40:40.9540331Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:40.9541016Z (EngineCore_DP0 pid=2813) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:40.9541736Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:40.9542531Z (EngineCore_DP0 pid=2813) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:40.9543021Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9543663Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:40.9544250Z (EngineCore_DP0 pid=2813) return func(*args, **kwargs) 2025-10-10T01:40:40.9544631Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9545266Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:40.9545887Z (EngineCore_DP0 pid=2813) worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:40.9546310Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9547064Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:40.9547730Z (EngineCore_DP0 pid=2813) module = importlib.import_module(module_name) 2025-10-10T01:40:40.9548172Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9548747Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:40.9549377Z (EngineCore_DP0 pid=2813) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:40.9549871Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9550368Z (EngineCore_DP0 pid=2813) File "", line 1387, in _gcd_import 2025-10-10T01:40:40.9550923Z (EngineCore_DP0 pid=2813) File "", line 1360, in _find_and_load 2025-10-10T01:40:40.9551502Z (EngineCore_DP0 pid=2813) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:40.9552079Z (EngineCore_DP0 pid=2813) File "", line 935, in _load_unlocked 2025-10-10T01:40:40.9552645Z (EngineCore_DP0 pid=2813) File "", line 999, in exec_module 2025-10-10T01:40:40.9553256Z (EngineCore_DP0 pid=2813) File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:40.9554045Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:40.9554730Z (EngineCore_DP0 pid=2813) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:40.9555455Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:40.9556186Z (EngineCore_DP0 pid=2813) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:40.9556960Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:40.9557634Z (EngineCore_DP0 pid=2813) class FlashAttentionMetadataBuilder( 2025-10-10T01:40:40.9558426Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:40.9559284Z (EngineCore_DP0 pid=2813) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:40.9559829Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9560525Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:40.9561210Z (EngineCore_DP0 pid=2813) if not is_fa_version_supported(fa_version): 2025-10-10T01:40:40.9561648Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9562383Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:40.9563087Z (EngineCore_DP0 pid=2813) return _is_fa2_supported(device)[0] 2025-10-10T01:40:40.9563500Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9564280Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:40.9565007Z (EngineCore_DP0 pid=2813) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:40.9565457Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9566125Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:40.9566769Z (EngineCore_DP0 pid=2813) prop = get_device_properties(device) 2025-10-10T01:40:40.9567186Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:40.9567837Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:40.9568493Z (EngineCore_DP0 pid=2813) _lazy_init() # will define _get_device_properties 2025-10-10T01:40:40.9568881Z (EngineCore_DP0 pid=2813) ^^^^^^^^^^^^ 2025-10-10T01:40:40.9569459Z (EngineCore_DP0 pid=2813) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:40.9570013Z (EngineCore_DP0 pid=2813) raise RuntimeError( 2025-10-10T01:40:40.9570705Z (EngineCore_DP0 pid=2813) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:41.3482177Z FAILED 2025-10-10T01:40:41.3609516Z models/test_initialization.py::test_can_initialize_large_subset[TarsierForConditionalGeneration] Fork a new process to run a test 2817 2025-10-10T01:40:41.3619327Z Fork a new process to run a test 0 2025-10-10T01:40:41.3898054Z INFO 10-10 01:40:41 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='TarsierForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'omni-research/Tarsier-7b'} 2025-10-10T01:40:41.5367495Z 2025-10-10T01:40:41.5369449Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:40:41.5369889Z config.json: 1.17kB [00:00, 6.39MB/s] 2025-10-10T01:40:41.6518928Z 2025-10-10T01:40:41.6520008Z preprocessor_config.json: 0% 0.00/505 [00:00", line 1387, in _gcd_import 2025-10-10T01:40:49.6762523Z 2025-10-10T01:40:49.6762775Z ERROR 10-10 01:40:49 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:40:49.6763130Z 2025-10-10T01:40:49.6763421Z ERROR 10-10 01:40:49 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:49.6763807Z 2025-10-10T01:40:49.6764014Z ERROR 10-10 01:40:49 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:40:49.6764284Z 2025-10-10T01:40:49.6764514Z ERROR 10-10 01:40:49 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:40:49.6764808Z 2025-10-10T01:40:49.6765036Z ERROR 10-10 01:40:49 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:49.6765345Z 2025-10-10T01:40:49.6765744Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/tarsier.py", line 24, in 2025-10-10T01:40:49.6766261Z 2025-10-10T01:40:49.6766490Z ERROR 10-10 01:40:49 [registry.py:542] from vllm.model_executor.models.llava import LlavaDummyInputsBuilder 2025-10-10T01:40:49.6766796Z 2025-10-10T01:40:49.6767306Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava.py", line 40, in 2025-10-10T01:40:49.6767937Z 2025-10-10T01:40:49.6768244Z ERROR 10-10 01:40:49 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T01:40:49.6768730Z 2025-10-10T01:40:49.6769150Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:40:49.6769597Z 2025-10-10T01:40:49.6769749Z ERROR 10-10 01:40:49 [registry.py:542] from xformers import ops as xops 2025-10-10T01:40:49.6769969Z 2025-10-10T01:40:49.6770292Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:40:49.6770686Z 2025-10-10T01:40:49.6771432Z ERROR 10-10 01:40:49 [registry.py:542] from .fmha import ( 2025-10-10T01:40:49.6771659Z 2025-10-10T01:40:49.6771993Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:40:49.6772405Z 2025-10-10T01:40:49.6772637Z ERROR 10-10 01:40:49 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:40:49.6772938Z 2025-10-10T01:40:49.6773292Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:40:49.6773738Z 2025-10-10T01:40:49.6773895Z ERROR 10-10 01:40:49 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:40:49.6774127Z 2025-10-10T01:40:49.6774271Z ERROR 10-10 01:40:49 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:49.6774484Z 2025-10-10T01:40:49.6774808Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:40:49.6775208Z 2025-10-10T01:40:49.6775318Z ERROR 10-10 01:40:49 [registry.py:542] value = func() 2025-10-10T01:40:49.6775508Z 2025-10-10T01:40:49.6775619Z ERROR 10-10 01:40:49 [registry.py:542] ^^^^^^ 2025-10-10T01:40:49.6775802Z 2025-10-10T01:40:49.6776155Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:40:49.6776627Z 2025-10-10T01:40:49.6776816Z ERROR 10-10 01:40:49 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:40:49.6777075Z 2025-10-10T01:40:49.6777219Z ERROR 10-10 01:40:49 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:49.6777435Z 2025-10-10T01:40:49.6777798Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:49.6778224Z 2025-10-10T01:40:49.6778375Z ERROR 10-10 01:40:49 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:40:49.6778602Z 2025-10-10T01:40:49.6778731Z ERROR 10-10 01:40:49 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:49.6778941Z 2025-10-10T01:40:49.6779286Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:49.6779714Z 2025-10-10T01:40:49.6779882Z ERROR 10-10 01:40:49 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:49.6780127Z 2025-10-10T01:40:49.6780232Z ERROR 10-10 01:40:49 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:40:49.6780401Z 2025-10-10T01:40:49.6780787Z ERROR 10-10 01:40:49 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:49.6781175Z 2025-10-10T01:40:49.6781296Z ERROR 10-10 01:40:49 [registry.py:542] raise RuntimeError( 2025-10-10T01:40:49.6781482Z 2025-10-10T01:40:49.6781875Z ERROR 10-10 01:40:49 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:50.0520439Z FAILED 2025-10-10T01:40:50.0647694Z models/test_initialization.py::test_can_initialize_large_subset[Eagle3LlamaForCausalLM] Fork a new process to run a test 2893 2025-10-10T01:40:50.0658713Z Fork a new process to run a test 0 2025-10-10T01:40:50.0933552Z INFO 10-10 01:40:50 [utils.py:233] non-default args: {'tokenizer': 'meta-llama/Llama-3.1-8B-Instruct', 'trust_remote_code': True, 'load_format': 'dummy', 'max_model_len': 10240, 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Eagle3LlamaForCausalLM', exist_overrides={}, use_original_num_layers=True), 'speculative_config': {'model': 'yuhuili/EAGLE3-LLaMA3.1-Instruct-8B', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'meta-llama/Llama-3.1-8B-Instruct'} 2025-10-10T01:40:50.2057031Z 2025-10-10T01:40:50.2058101Z config.json: 0% 0.00/855 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:40:51.8923933Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] EngineCore failed to start. 2025-10-10T01:40:51.8924455Z 2025-10-10T01:40:51.8925001Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] Traceback (most recent call last): 2025-10-10T01:40:51.8925388Z 2025-10-10T01:40:51.8926066Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:51.8926549Z 2025-10-10T01:40:51.8926858Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:51.8927170Z 2025-10-10T01:40:51.8927446Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8927944Z 2025-10-10T01:40:51.8928433Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:51.8928907Z 2025-10-10T01:40:51.8929259Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:51.8929611Z 2025-10-10T01:40:51.8930245Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:51.8930824Z 2025-10-10T01:40:51.8931168Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:51.8931488Z 2025-10-10T01:40:51.8931776Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8932063Z 2025-10-10T01:40:51.8932569Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:51.8933156Z 2025-10-10T01:40:51.8933534Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] self._init_executor() 2025-10-10T01:40:51.8933813Z 2025-10-10T01:40:51.8934490Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:51.8935230Z 2025-10-10T01:40:51.8935677Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:51.8936121Z 2025-10-10T01:40:51.8936699Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:51.8937421Z 2025-10-10T01:40:51.8937797Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:51.8938241Z 2025-10-10T01:40:51.8938721Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8939026Z 2025-10-10T01:40:51.8939530Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:51.8940019Z 2025-10-10T01:40:51.8940290Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:40:51.8940566Z 2025-10-10T01:40:51.8940816Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8941075Z 2025-10-10T01:40:51.8941597Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:51.8942072Z 2025-10-10T01:40:51.8942367Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:51.8942655Z 2025-10-10T01:40:51.8942925Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8943192Z 2025-10-10T01:40:51.8943720Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:51.8944213Z 2025-10-10T01:40:51.8944511Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:40:51.8944865Z 2025-10-10T01:40:51.8945139Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8945418Z 2025-10-10T01:40:51.8945873Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:51.8946289Z 2025-10-10T01:40:51.8946626Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:51.8946955Z 2025-10-10T01:40:51.8947255Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8947536Z 2025-10-10T01:40:51.8947881Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:40:51.8948210Z 2025-10-10T01:40:51.8948557Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:40:51.8948944Z 2025-10-10T01:40:51.8949320Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:51.8949682Z 2025-10-10T01:40:51.8950021Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:40:51.8950356Z 2025-10-10T01:40:51.8950715Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:40:51.8951069Z 2025-10-10T01:40:51.8951438Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:51.8951799Z 2025-10-10T01:40:51.8952292Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:51.8952754Z 2025-10-10T01:40:51.8953179Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:51.8953513Z 2025-10-10T01:40:51.8954041Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:51.8954520Z 2025-10-10T01:40:51.8954895Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:51.8955256Z 2025-10-10T01:40:51.8955840Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:51.8956346Z 2025-10-10T01:40:51.8956636Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:40:51.8956939Z 2025-10-10T01:40:51.8957564Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:51.8958141Z 2025-10-10T01:40:51.8958505Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:51.8958860Z 2025-10-10T01:40:51.8959254Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8959577Z 2025-10-10T01:40:51.8960145Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:51.8960664Z 2025-10-10T01:40:51.8960968Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:40:51.8961264Z 2025-10-10T01:40:51.8961541Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8961811Z 2025-10-10T01:40:51.8962422Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:51.8962985Z 2025-10-10T01:40:51.8963271Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:40:51.8963558Z 2025-10-10T01:40:51.8963817Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8964088Z 2025-10-10T01:40:51.8964736Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:51.8965277Z 2025-10-10T01:40:51.8965595Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:51.8965918Z 2025-10-10T01:40:51.8966188Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8966458Z 2025-10-10T01:40:51.8966982Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:51.8967465Z 2025-10-10T01:40:51.8967750Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:40:51.8968033Z 2025-10-10T01:40:51.8968397Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8968671Z 2025-10-10T01:40:51.8969193Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:51.8969673Z 2025-10-10T01:40:51.8969979Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:40:51.8970286Z 2025-10-10T01:40:51.8970507Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:40:51.8970755Z 2025-10-10T01:40:51.8971235Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:51.8971690Z 2025-10-10T01:40:51.8971931Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] raise RuntimeError( 2025-10-10T01:40:51.8972186Z 2025-10-10T01:40:51.8972774Z (EngineCore_DP0 pid=2901) ERROR 10-10 01:40:51 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:51.8973491Z (EngineCore_DP0 pid=2901) Process EngineCore_DP0: 2025-10-10T01:40:51.8973884Z (EngineCore_DP0 pid=2901) Traceback (most recent call last): 2025-10-10T01:40:51.8974484Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:40:51.8975041Z (EngineCore_DP0 pid=2901) self.run() 2025-10-10T01:40:51.8975565Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:40:51.8976139Z (EngineCore_DP0 pid=2901) self._target(*self._args, **self._kwargs) 2025-10-10T01:40:51.8976807Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:40:51.8977354Z (EngineCore_DP0 pid=2901) raise e 2025-10-10T01:40:51.8977932Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:40:51.8978565Z (EngineCore_DP0 pid=2901) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:40:51.8979005Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8979629Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:40:51.8980270Z (EngineCore_DP0 pid=2901) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:40:51.8980988Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:40:51.8981606Z (EngineCore_DP0 pid=2901) self.model_executor = executor_class(vllm_config) 2025-10-10T01:40:51.8982054Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8982705Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:40:51.8983287Z (EngineCore_DP0 pid=2901) self._init_executor() 2025-10-10T01:40:51.8983946Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:40:51.8984632Z (EngineCore_DP0 pid=2901) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:40:51.8985431Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:40:51.8986148Z (EngineCore_DP0 pid=2901) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:40:51.8986644Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8987286Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:40:51.8987871Z (EngineCore_DP0 pid=2901) return func(*args, **kwargs) 2025-10-10T01:40:51.8988254Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8988891Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:40:51.8989539Z (EngineCore_DP0 pid=2901) worker_class = resolve_obj_by_qualname( 2025-10-10T01:40:51.8989956Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8990624Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:40:51.8991477Z (EngineCore_DP0 pid=2901) module = importlib.import_module(module_name) 2025-10-10T01:40:51.8991932Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8992584Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:40:51.8993213Z (EngineCore_DP0 pid=2901) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:40:51.8993702Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.8994207Z (EngineCore_DP0 pid=2901) File "", line 1387, in _gcd_import 2025-10-10T01:40:51.8994762Z (EngineCore_DP0 pid=2901) File "", line 1360, in _find_and_load 2025-10-10T01:40:51.8995350Z (EngineCore_DP0 pid=2901) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:40:51.8995930Z (EngineCore_DP0 pid=2901) File "", line 935, in _load_unlocked 2025-10-10T01:40:51.8996785Z (EngineCore_DP0 pid=2901) File "", line 999, in exec_module 2025-10-10T01:40:51.8997398Z (EngineCore_DP0 pid=2901) File "", line 488, in _call_with_frames_removed 2025-10-10T01:40:51.8998131Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:40:51.8998923Z (EngineCore_DP0 pid=2901) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:40:51.8999758Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:40:51.9000485Z (EngineCore_DP0 pid=2901) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:40:51.9001273Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:40:51.9001949Z (EngineCore_DP0 pid=2901) class FlashAttentionMetadataBuilder( 2025-10-10T01:40:51.9002742Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:40:51.9003687Z (EngineCore_DP0 pid=2901) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:40:51.9004179Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.9004867Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:40:51.9005550Z (EngineCore_DP0 pid=2901) if not is_fa_version_supported(fa_version): 2025-10-10T01:40:51.9005991Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.9006735Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:40:51.9007434Z (EngineCore_DP0 pid=2901) return _is_fa2_supported(device)[0] 2025-10-10T01:40:51.9007843Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.9008574Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:40:51.9009290Z (EngineCore_DP0 pid=2901) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:40:51.9009745Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.9010419Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:40:51.9011134Z (EngineCore_DP0 pid=2901) prop = get_device_properties(device) 2025-10-10T01:40:51.9011548Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:40:51.9012208Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:40:51.9012880Z (EngineCore_DP0 pid=2901) _lazy_init() # will define _get_device_properties 2025-10-10T01:40:51.9013272Z (EngineCore_DP0 pid=2901) ^^^^^^^^^^^^ 2025-10-10T01:40:51.9013848Z (EngineCore_DP0 pid=2901) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:40:51.9014412Z (EngineCore_DP0 pid=2901) raise RuntimeError( 2025-10-10T01:40:51.9015095Z (EngineCore_DP0 pid=2901) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:40:52.2936898Z FAILED 2025-10-10T01:40:52.3064161Z models/test_initialization.py::test_can_initialize_large_subset[CohereForCausalLM] Fork a new process to run a test 2905 2025-10-10T01:40:52.3074045Z Fork a new process to run a test 0 2025-10-10T01:40:52.3348984Z INFO 10-10 01:40:52 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='CohereForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'CohereForAI/c4ai-command-r-v01'} 2025-10-10T01:40:52.5648802Z 2025-10-10T01:40:52.5650437Z config.json: 0% 0.00/765 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:01.3767002Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:01.3767634Z 2025-10-10T01:41:01.3768184Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:01.3768589Z 2025-10-10T01:41:01.3769394Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:01.3770163Z 2025-10-10T01:41:01.3770701Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:01.3771099Z 2025-10-10T01:41:01.3771936Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3772422Z 2025-10-10T01:41:01.3773449Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:01.3774150Z 2025-10-10T01:41:01.3774608Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:01.3775063Z 2025-10-10T01:41:01.3775607Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:01.3776111Z 2025-10-10T01:41:01.3776495Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:01.3776872Z 2025-10-10T01:41:01.3777172Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3777587Z 2025-10-10T01:41:01.3778324Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:01.3778806Z 2025-10-10T01:41:01.3779169Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] self._init_executor() 2025-10-10T01:41:01.3779596Z 2025-10-10T01:41:01.3780358Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:01.3781006Z 2025-10-10T01:41:01.3781472Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:01.3781811Z 2025-10-10T01:41:01.3782485Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:01.3783093Z 2025-10-10T01:41:01.3783455Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:01.3783803Z 2025-10-10T01:41:01.3784094Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3784397Z 2025-10-10T01:41:01.3784889Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:01.3785477Z 2025-10-10T01:41:01.3785756Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:01.3786027Z 2025-10-10T01:41:01.3786298Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3786558Z 2025-10-10T01:41:01.3787080Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:01.3787554Z 2025-10-10T01:41:01.3787847Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:01.3788152Z 2025-10-10T01:41:01.3788427Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3788708Z 2025-10-10T01:41:01.3789337Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:01.3789836Z 2025-10-10T01:41:01.3790142Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:01.3790446Z 2025-10-10T01:41:01.3790715Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3790984Z 2025-10-10T01:41:01.3791437Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:01.3791852Z 2025-10-10T01:41:01.3792198Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:01.3792525Z 2025-10-10T01:41:01.3792819Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3793103Z 2025-10-10T01:41:01.3793445Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:01.3793782Z 2025-10-10T01:41:01.3794129Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:01.3794468Z 2025-10-10T01:41:01.3794840Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:01.3795255Z 2025-10-10T01:41:01.3795595Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:01.3795927Z 2025-10-10T01:41:01.3796649Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:01.3797022Z 2025-10-10T01:41:01.3797416Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:01.3797776Z 2025-10-10T01:41:01.3798281Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:01.3798755Z 2025-10-10T01:41:01.3799240Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:01.3799579Z 2025-10-10T01:41:01.3800100Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:01.3800675Z 2025-10-10T01:41:01.3801052Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:01.3801420Z 2025-10-10T01:41:01.3801969Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:01.3802478Z 2025-10-10T01:41:01.3802767Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:01.3803064Z 2025-10-10T01:41:01.3803690Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:01.3804267Z 2025-10-10T01:41:01.3804627Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:01.3804976Z 2025-10-10T01:41:01.3805379Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3805640Z 2025-10-10T01:41:01.3806207Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:01.3806720Z 2025-10-10T01:41:01.3807017Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:01.3807309Z 2025-10-10T01:41:01.3807582Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3807863Z 2025-10-10T01:41:01.3808462Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:01.3809028Z 2025-10-10T01:41:01.3809306Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:01.3809593Z 2025-10-10T01:41:01.3809849Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3810114Z 2025-10-10T01:41:01.3810688Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:01.3811286Z 2025-10-10T01:41:01.3811601Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:01.3811909Z 2025-10-10T01:41:01.3812186Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3812456Z 2025-10-10T01:41:01.3812986Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:01.3813463Z 2025-10-10T01:41:01.3813748Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:01.3814035Z 2025-10-10T01:41:01.3814292Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3814566Z 2025-10-10T01:41:01.3815077Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:01.3815563Z 2025-10-10T01:41:01.3815917Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:01.3816231Z 2025-10-10T01:41:01.3816466Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:01.3816706Z 2025-10-10T01:41:01.3817200Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:01.3817645Z 2025-10-10T01:41:01.3817892Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] raise RuntimeError( 2025-10-10T01:41:01.3818152Z 2025-10-10T01:41:01.3818723Z (EngineCore_DP0 pid=2983) ERROR 10-10 01:41:01 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:01.3819433Z (EngineCore_DP0 pid=2983) Process EngineCore_DP0: 2025-10-10T01:41:01.3819831Z (EngineCore_DP0 pid=2983) Traceback (most recent call last): 2025-10-10T01:41:01.3820520Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:01.3821044Z (EngineCore_DP0 pid=2983) self.run() 2025-10-10T01:41:01.3821580Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:01.3822150Z (EngineCore_DP0 pid=2983) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:01.3822833Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:01.3823379Z (EngineCore_DP0 pid=2983) raise e 2025-10-10T01:41:01.3823966Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:01.3824606Z (EngineCore_DP0 pid=2983) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:01.3825046Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3825667Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:01.3826298Z (EngineCore_DP0 pid=2983) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:01.3826956Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:01.3827633Z (EngineCore_DP0 pid=2983) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:01.3828083Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3828738Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:01.3829315Z (EngineCore_DP0 pid=2983) self._init_executor() 2025-10-10T01:41:01.3829975Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:01.3830673Z (EngineCore_DP0 pid=2983) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:01.3831395Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:01.3832103Z (EngineCore_DP0 pid=2983) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:01.3832589Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3833285Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:01.3833871Z (EngineCore_DP0 pid=2983) return func(*args, **kwargs) 2025-10-10T01:41:01.3834256Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3834918Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:01.3835552Z (EngineCore_DP0 pid=2983) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:01.3835982Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3836671Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:01.3837340Z (EngineCore_DP0 pid=2983) module = importlib.import_module(module_name) 2025-10-10T01:41:01.3837860Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3838448Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:01.3839166Z (EngineCore_DP0 pid=2983) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:01.3839659Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3840155Z (EngineCore_DP0 pid=2983) File "", line 1387, in _gcd_import 2025-10-10T01:41:01.3840712Z (EngineCore_DP0 pid=2983) File "", line 1360, in _find_and_load 2025-10-10T01:41:01.3841302Z (EngineCore_DP0 pid=2983) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:01.3841884Z (EngineCore_DP0 pid=2983) File "", line 935, in _load_unlocked 2025-10-10T01:41:01.3842444Z (EngineCore_DP0 pid=2983) File "", line 999, in exec_module 2025-10-10T01:41:01.3843044Z (EngineCore_DP0 pid=2983) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:01.3843772Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:01.3844443Z (EngineCore_DP0 pid=2983) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:01.3845206Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:01.3845920Z (EngineCore_DP0 pid=2983) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:01.3846693Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:01.3847357Z (EngineCore_DP0 pid=2983) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:01.3848139Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:01.3848958Z (EngineCore_DP0 pid=2983) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:01.3849440Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3850120Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:01.3850846Z (EngineCore_DP0 pid=2983) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:01.3851287Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3852024Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:01.3852761Z (EngineCore_DP0 pid=2983) return _is_fa2_supported(device)[0] 2025-10-10T01:41:01.3853186Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3853907Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:01.3854625Z (EngineCore_DP0 pid=2983) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:01.3855084Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3855829Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:01.3856474Z (EngineCore_DP0 pid=2983) prop = get_device_properties(device) 2025-10-10T01:41:01.3856883Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:01.3857540Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:01.3858211Z (EngineCore_DP0 pid=2983) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:01.3858603Z (EngineCore_DP0 pid=2983) ^^^^^^^^^^^^ 2025-10-10T01:41:01.3859183Z (EngineCore_DP0 pid=2983) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:01.3859750Z (EngineCore_DP0 pid=2983) raise RuntimeError( 2025-10-10T01:41:01.3860456Z (EngineCore_DP0 pid=2983) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:01.7867798Z FAILED 2025-10-10T01:41:01.7995049Z models/test_initialization.py::test_can_initialize_large_subset[MiniMaxM1ForCausalLM] Fork a new process to run a test 2987 2025-10-10T01:41:01.8005742Z Fork a new process to run a test 0 2025-10-10T01:41:01.8277399Z INFO 10-10 01:41:01 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiniMaxM1ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'MiniMaxAI/MiniMax-M1-40k'} 2025-10-10T01:41:01.9989795Z 2025-10-10T01:41:01.9991750Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:01.9992079Z config.json: 1.87kB [00:00, 10.3MB/s] 2025-10-10T01:41:02.0978033Z 2025-10-10T01:41:02.0979935Z configuration_minimax_m1.py: 0.00B [00:00, ?B/s] 2025-10-10T01:41:02.0980345Z configuration_minimax_m1.py: 7.30kB [00:00, 53.7MB/s] 2025-10-10T01:41:02.1062920Z A new version of the following files was downloaded from https://huggingface.co/MiniMaxAI/MiniMax-M1-40k: 2025-10-10T01:41:02.1063447Z - configuration_minimax_m1.py 2025-10-10T01:41:02.1064069Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:41:02.1399959Z You are using a model of type minimax_m1 to instantiate a model of type MiniMaxM1. This is not supported for all configurations of models and can yield errors. 2025-10-10T01:41:08.8555378Z INFO 10-10 01:41:08 [model.py:551] Resolved architecture: MiniMaxM1ForCausalLM 2025-10-10T01:41:08.8556354Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:41:08.9654943Z 2025-10-10T01:41:08.9676144Z model.safetensors.index.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:08.9676726Z model.safetensors.index.json: 823kB [00:00, 399MB/s] 2025-10-10T01:41:08.9967177Z 2025-10-10T01:41:09.1709815Z Parse safetensors files: 0% 0/413 [00:00= mamba page size. 2025-10-10T01:41:10.6777274Z 2025-10-10T01:41:10.6779073Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:10.6779414Z tokenizer_config.json: 1.97kB [00:00, 17.3MB/s] 2025-10-10T01:41:10.7562589Z 2025-10-10T01:41:10.7766336Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:10.7766647Z vocab.json: 4.71MB [00:00, 231MB/s] 2025-10-10T01:41:10.8352905Z 2025-10-10T01:41:10.8485697Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:41:10.8486242Z merges.txt: 2.41MB [00:00, 181MB/s] 2025-10-10T01:41:10.8860114Z 2025-10-10T01:41:10.9318129Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:10.9318456Z tokenizer.json: 9.73MB [00:00, 212MB/s] 2025-10-10T01:41:11.8357718Z You are using a model of type minimax_m1 to instantiate a model of type MiniMaxM1. This is not supported for all configurations of models and can yield errors. 2025-10-10T01:41:11.8559549Z (EngineCore_DP0 pid=3065) INFO 10-10 01:41:11 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:41:11.8640051Z (EngineCore_DP0 pid=3065) INFO 10-10 01:41:11 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='MiniMaxAI/MiniMax-M1-40k', speculative_config=None, tokenizer='MiniMaxAI/MiniMax-M1-40k', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=10240000, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=MiniMaxAI/MiniMax-M1-40k, enable_prefix_caching=False, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:11.9669507Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:11.9670150Z 2025-10-10T01:41:11.9670644Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:11.9671053Z 2025-10-10T01:41:11.9671774Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:11.9672383Z 2025-10-10T01:41:11.9672764Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:11.9673162Z 2025-10-10T01:41:11.9673525Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9673867Z 2025-10-10T01:41:11.9674428Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:11.9674871Z 2025-10-10T01:41:11.9675206Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:11.9675793Z 2025-10-10T01:41:11.9676273Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:11.9676708Z 2025-10-10T01:41:11.9677028Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:11.9677344Z 2025-10-10T01:41:11.9677630Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9677907Z 2025-10-10T01:41:11.9678409Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:11.9679134Z 2025-10-10T01:41:11.9679454Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] self._init_executor() 2025-10-10T01:41:11.9679941Z 2025-10-10T01:41:11.9680966Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:11.9681884Z 2025-10-10T01:41:11.9682596Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:11.9682952Z 2025-10-10T01:41:11.9683519Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:11.9684023Z 2025-10-10T01:41:11.9684386Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:11.9684718Z 2025-10-10T01:41:11.9685010Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9685291Z 2025-10-10T01:41:11.9685780Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:11.9686250Z 2025-10-10T01:41:11.9686696Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:11.9686990Z 2025-10-10T01:41:11.9687246Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9687508Z 2025-10-10T01:41:11.9688034Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:11.9688513Z 2025-10-10T01:41:11.9688800Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:11.9689087Z 2025-10-10T01:41:11.9689370Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9689644Z 2025-10-10T01:41:11.9690202Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:11.9690707Z 2025-10-10T01:41:11.9691029Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:11.9691335Z 2025-10-10T01:41:11.9691604Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9691883Z 2025-10-10T01:41:11.9692317Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:11.9692791Z 2025-10-10T01:41:11.9693131Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:11.9693465Z 2025-10-10T01:41:11.9693752Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9694036Z 2025-10-10T01:41:11.9694375Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:11.9694702Z 2025-10-10T01:41:11.9695050Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:11.9695382Z 2025-10-10T01:41:11.9695760Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:11.9696400Z 2025-10-10T01:41:11.9696826Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:11.9697177Z 2025-10-10T01:41:11.9697553Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:11.9698141Z 2025-10-10T01:41:11.9698638Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:11.9699015Z 2025-10-10T01:41:11.9699525Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:11.9699995Z 2025-10-10T01:41:11.9700349Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:11.9700710Z 2025-10-10T01:41:11.9701244Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:11.9701728Z 2025-10-10T01:41:11.9702248Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:11.9702621Z 2025-10-10T01:41:11.9703198Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:11.9703703Z 2025-10-10T01:41:11.9704009Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:11.9704304Z 2025-10-10T01:41:11.9704928Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:11.9705502Z 2025-10-10T01:41:11.9705865Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:11.9706222Z 2025-10-10T01:41:11.9706477Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9706742Z 2025-10-10T01:41:11.9707307Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:11.9707824Z 2025-10-10T01:41:11.9708119Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:11.9708474Z 2025-10-10T01:41:11.9708753Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9709025Z 2025-10-10T01:41:11.9709639Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:11.9710208Z 2025-10-10T01:41:11.9710494Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:11.9710775Z 2025-10-10T01:41:11.9711038Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9711301Z 2025-10-10T01:41:11.9711880Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:11.9712416Z 2025-10-10T01:41:11.9712733Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:11.9713047Z 2025-10-10T01:41:11.9713365Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9713640Z 2025-10-10T01:41:11.9714162Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:11.9714649Z 2025-10-10T01:41:11.9714928Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:11.9715209Z 2025-10-10T01:41:11.9715473Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9715753Z 2025-10-10T01:41:11.9716276Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:11.9716754Z 2025-10-10T01:41:11.9717067Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:11.9717468Z 2025-10-10T01:41:11.9717699Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:11.9717944Z 2025-10-10T01:41:11.9718424Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:11.9718885Z 2025-10-10T01:41:11.9719221Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] raise RuntimeError( 2025-10-10T01:41:11.9719483Z 2025-10-10T01:41:11.9720048Z (EngineCore_DP0 pid=3065) ERROR 10-10 01:41:11 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:11.9720743Z (EngineCore_DP0 pid=3065) Process EngineCore_DP0: 2025-10-10T01:41:11.9721176Z (EngineCore_DP0 pid=3065) Traceback (most recent call last): 2025-10-10T01:41:11.9721787Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:11.9722295Z (EngineCore_DP0 pid=3065) self.run() 2025-10-10T01:41:11.9722820Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:11.9723383Z (EngineCore_DP0 pid=3065) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:11.9724051Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:11.9724679Z (EngineCore_DP0 pid=3065) raise e 2025-10-10T01:41:11.9725262Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:11.9725903Z (EngineCore_DP0 pid=3065) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:11.9726343Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9726964Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:11.9727622Z (EngineCore_DP0 pid=3065) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:11.9728274Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:11.9728894Z (EngineCore_DP0 pid=3065) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:11.9729348Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9730048Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:11.9730635Z (EngineCore_DP0 pid=3065) self._init_executor() 2025-10-10T01:41:11.9731303Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:11.9731984Z (EngineCore_DP0 pid=3065) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:11.9732710Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:11.9733436Z (EngineCore_DP0 pid=3065) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:11.9733939Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9734679Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:11.9735263Z (EngineCore_DP0 pid=3065) return func(*args, **kwargs) 2025-10-10T01:41:11.9735644Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9736279Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:11.9736922Z (EngineCore_DP0 pid=3065) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:11.9737347Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9738005Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:11.9738656Z (EngineCore_DP0 pid=3065) module = importlib.import_module(module_name) 2025-10-10T01:41:11.9739098Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9739677Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:11.9740312Z (EngineCore_DP0 pid=3065) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:11.9740808Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9741356Z (EngineCore_DP0 pid=3065) File "", line 1387, in _gcd_import 2025-10-10T01:41:11.9741909Z (EngineCore_DP0 pid=3065) File "", line 1360, in _find_and_load 2025-10-10T01:41:11.9742493Z (EngineCore_DP0 pid=3065) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:11.9743077Z (EngineCore_DP0 pid=3065) File "", line 935, in _load_unlocked 2025-10-10T01:41:11.9743645Z (EngineCore_DP0 pid=3065) File "", line 999, in exec_module 2025-10-10T01:41:11.9744246Z (EngineCore_DP0 pid=3065) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:11.9744983Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:11.9745691Z (EngineCore_DP0 pid=3065) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:11.9746444Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:11.9747180Z (EngineCore_DP0 pid=3065) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:11.9748720Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:11.9749399Z (EngineCore_DP0 pid=3065) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:11.9750191Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:11.9751009Z (EngineCore_DP0 pid=3065) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:11.9751498Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9752191Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:11.9752879Z (EngineCore_DP0 pid=3065) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:11.9753402Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9754199Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:11.9754913Z (EngineCore_DP0 pid=3065) return _is_fa2_supported(device)[0] 2025-10-10T01:41:11.9755329Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9756062Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:11.9756787Z (EngineCore_DP0 pid=3065) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:11.9757251Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9757935Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:11.9758577Z (EngineCore_DP0 pid=3065) prop = get_device_properties(device) 2025-10-10T01:41:11.9758993Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:11.9759712Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:11.9760433Z (EngineCore_DP0 pid=3065) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:11.9760831Z (EngineCore_DP0 pid=3065) ^^^^^^^^^^^^ 2025-10-10T01:41:11.9761417Z (EngineCore_DP0 pid=3065) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:11.9761981Z (EngineCore_DP0 pid=3065) raise RuntimeError( 2025-10-10T01:41:11.9762667Z (EngineCore_DP0 pid=3065) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:12.3753688Z FAILED 2025-10-10T01:41:12.3880820Z models/test_initialization.py::test_can_initialize_large_subset[OlmoeForCausalLM] Fork a new process to run a test 3069 2025-10-10T01:41:12.3891792Z Fork a new process to run a test 0 2025-10-10T01:41:12.4168520Z INFO 10-10 01:41:12 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='OlmoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'allenai/OLMoE-1B-7B-0924-Instruct'} 2025-10-10T01:41:12.6226630Z 2025-10-10T01:41:12.6227429Z config.json: 0% 0.00/759 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:20.2575155Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:20.2575792Z 2025-10-10T01:41:20.2576211Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:20.2576564Z 2025-10-10T01:41:20.2577249Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:20.2578062Z 2025-10-10T01:41:20.2578469Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:20.2578857Z 2025-10-10T01:41:20.2579208Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2579559Z 2025-10-10T01:41:20.2580169Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:20.2580799Z 2025-10-10T01:41:20.2581482Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:20.2581864Z 2025-10-10T01:41:20.2582559Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:20.2583020Z 2025-10-10T01:41:20.2583341Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:20.2583657Z 2025-10-10T01:41:20.2583937Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2584211Z 2025-10-10T01:41:20.2584953Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:20.2585804Z 2025-10-10T01:41:20.2586336Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] self._init_executor() 2025-10-10T01:41:20.2586799Z 2025-10-10T01:41:20.2587846Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:20.2588780Z 2025-10-10T01:41:20.2589260Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:20.2589608Z 2025-10-10T01:41:20.2590182Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:20.2590711Z 2025-10-10T01:41:20.2591185Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:20.2591531Z 2025-10-10T01:41:20.2591825Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2592120Z 2025-10-10T01:41:20.2592626Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:20.2593081Z 2025-10-10T01:41:20.2593352Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:20.2593622Z 2025-10-10T01:41:20.2593874Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2594133Z 2025-10-10T01:41:20.2594654Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:20.2595135Z 2025-10-10T01:41:20.2595428Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:20.2595777Z 2025-10-10T01:41:20.2596046Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2596602Z 2025-10-10T01:41:20.2597154Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:20.2597651Z 2025-10-10T01:41:20.2597954Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:20.2598258Z 2025-10-10T01:41:20.2598533Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2598812Z 2025-10-10T01:41:20.2599379Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:20.2599795Z 2025-10-10T01:41:20.2600337Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:20.2600687Z 2025-10-10T01:41:20.2600982Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2601263Z 2025-10-10T01:41:20.2601607Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:20.2601937Z 2025-10-10T01:41:20.2602292Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:20.2602632Z 2025-10-10T01:41:20.2603012Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:20.2603370Z 2025-10-10T01:41:20.2603727Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:20.2604065Z 2025-10-10T01:41:20.2604426Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:20.2604781Z 2025-10-10T01:41:20.2605151Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:20.2605517Z 2025-10-10T01:41:20.2606016Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:20.2606585Z 2025-10-10T01:41:20.2606939Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:20.2607286Z 2025-10-10T01:41:20.2607805Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:20.2608283Z 2025-10-10T01:41:20.2608655Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:20.2609012Z 2025-10-10T01:41:20.2609564Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:20.2610074Z 2025-10-10T01:41:20.2610371Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:20.2610674Z 2025-10-10T01:41:20.2611299Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:20.2611953Z 2025-10-10T01:41:20.2612318Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:20.2612674Z 2025-10-10T01:41:20.2612925Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2613191Z 2025-10-10T01:41:20.2613752Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:20.2614273Z 2025-10-10T01:41:20.2614568Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:20.2614867Z 2025-10-10T01:41:20.2615143Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2615493Z 2025-10-10T01:41:20.2616125Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:20.2616680Z 2025-10-10T01:41:20.2616966Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:20.2617248Z 2025-10-10T01:41:20.2617509Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2617776Z 2025-10-10T01:41:20.2618368Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:20.2618905Z 2025-10-10T01:41:20.2619221Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:20.2619536Z 2025-10-10T01:41:20.2619806Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2620083Z 2025-10-10T01:41:20.2620600Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:20.2621087Z 2025-10-10T01:41:20.2621424Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:20.2621707Z 2025-10-10T01:41:20.2621971Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2622237Z 2025-10-10T01:41:20.2622767Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:20.2623247Z 2025-10-10T01:41:20.2623561Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:20.2623866Z 2025-10-10T01:41:20.2624090Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:20.2624337Z 2025-10-10T01:41:20.2624821Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:20.2625279Z 2025-10-10T01:41:20.2625521Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] raise RuntimeError( 2025-10-10T01:41:20.2625783Z 2025-10-10T01:41:20.2626348Z (EngineCore_DP0 pid=3127) ERROR 10-10 01:41:20 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:20.2627098Z (EngineCore_DP0 pid=3127) Process EngineCore_DP0: 2025-10-10T01:41:20.2627497Z (EngineCore_DP0 pid=3127) Traceback (most recent call last): 2025-10-10T01:41:20.2628097Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:20.2628609Z (EngineCore_DP0 pid=3127) self.run() 2025-10-10T01:41:20.2629140Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:20.2629703Z (EngineCore_DP0 pid=3127) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:20.2630374Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:20.2631016Z (EngineCore_DP0 pid=3127) raise e 2025-10-10T01:41:20.2631615Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:20.2632247Z (EngineCore_DP0 pid=3127) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:20.2632691Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2633319Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:20.2633959Z (EngineCore_DP0 pid=3127) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:20.2634621Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:20.2635245Z (EngineCore_DP0 pid=3127) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:20.2635697Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2636355Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:20.2636933Z (EngineCore_DP0 pid=3127) self._init_executor() 2025-10-10T01:41:20.2637591Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:20.2638323Z (EngineCore_DP0 pid=3127) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:20.2639119Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:20.2639844Z (EngineCore_DP0 pid=3127) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:20.2640336Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2640970Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:20.2641556Z (EngineCore_DP0 pid=3127) return func(*args, **kwargs) 2025-10-10T01:41:20.2641937Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2649098Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:20.2649900Z (EngineCore_DP0 pid=3127) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:20.2650372Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2651191Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:20.2651874Z (EngineCore_DP0 pid=3127) module = importlib.import_module(module_name) 2025-10-10T01:41:20.2652323Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2652915Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:20.2653553Z (EngineCore_DP0 pid=3127) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:20.2654073Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2654579Z (EngineCore_DP0 pid=3127) File "", line 1387, in _gcd_import 2025-10-10T01:41:20.2655229Z (EngineCore_DP0 pid=3127) File "", line 1360, in _find_and_load 2025-10-10T01:41:20.2655840Z (EngineCore_DP0 pid=3127) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:20.2656425Z (EngineCore_DP0 pid=3127) File "", line 935, in _load_unlocked 2025-10-10T01:41:20.2656998Z (EngineCore_DP0 pid=3127) File "", line 999, in exec_module 2025-10-10T01:41:20.2657603Z (EngineCore_DP0 pid=3127) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:20.2658338Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:20.2659046Z (EngineCore_DP0 pid=3127) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:20.2659783Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:20.2660509Z (EngineCore_DP0 pid=3127) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:20.2661318Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:20.2661996Z (EngineCore_DP0 pid=3127) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:20.2662853Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:20.2663673Z (EngineCore_DP0 pid=3127) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:20.2664158Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2664866Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:20.2665553Z (EngineCore_DP0 pid=3127) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:20.2665991Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2666737Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:20.2667457Z (EngineCore_DP0 pid=3127) return _is_fa2_supported(device)[0] 2025-10-10T01:41:20.2667873Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2668594Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:20.2669363Z (EngineCore_DP0 pid=3127) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:20.2669817Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2670498Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:20.2671157Z (EngineCore_DP0 pid=3127) prop = get_device_properties(device) 2025-10-10T01:41:20.2671584Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:20.2672249Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:20.2672908Z (EngineCore_DP0 pid=3127) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:20.2673387Z (EngineCore_DP0 pid=3127) ^^^^^^^^^^^^ 2025-10-10T01:41:20.2673981Z (EngineCore_DP0 pid=3127) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:20.2674547Z (EngineCore_DP0 pid=3127) raise RuntimeError( 2025-10-10T01:41:20.2675237Z (EngineCore_DP0 pid=3127) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:20.6561518Z FAILED 2025-10-10T01:41:20.6688878Z models/test_initialization.py::test_can_initialize_large_subset[TransformersEmbeddingModel] Fork a new process to run a test 3131 2025-10-10T01:41:20.6700364Z Fork a new process to run a test 0 2025-10-10T01:41:20.6704451Z `transformers==4.56.2` installed, but `transformers>=4.57.0.dev0` is required to run this model. 2025-10-10T01:41:20.9710913Z PASSED 2025-10-10T01:41:20.9836150Z models/test_initialization.py::test_can_initialize_large_subset[HCXVisionForCausalLM] Fork a new process to run a test 3132 2025-10-10T01:41:20.9847642Z Fork a new process to run a test 0 2025-10-10T01:41:21.0125876Z INFO 10-10 01:41:21 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='HCXVisionForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B'} 2025-10-10T01:41:21.1972938Z 2025-10-10T01:41:21.1975324Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:21.1975641Z config.json: 5.51kB [00:00, 24.1MB/s] 2025-10-10T01:41:21.2808748Z 2025-10-10T01:41:21.2810848Z configuration_hyperclovax.py: 0.00B [00:00, ?B/s] 2025-10-10T01:41:21.2811275Z configuration_hyperclovax.py: 2.44kB [00:00, 10.4MB/s] 2025-10-10T01:41:21.2899724Z A new version of the following files was downloaded from https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B: 2025-10-10T01:41:21.2900415Z - configuration_hyperclovax.py 2025-10-10T01:41:21.2901039Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:41:21.5347485Z 2025-10-10T01:41:21.5348430Z preprocessor_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:21.5348830Z preprocessor_config.json: 1.76kB [00:00, 15.5MB/s] 2025-10-10T01:41:28.5080973Z INFO 10-10 01:41:28 [model.py:551] Resolved architecture: HCXVisionForCausalLM 2025-10-10T01:41:28.5081512Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:41:28.5322987Z INFO 10-10 01:41:28 [model.py:1545] Using max model len 131072 2025-10-10T01:41:28.5325531Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:41:28.5866832Z INFO 10-10 01:41:28 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:41:28.6241110Z 2025-10-10T01:41:28.6243316Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:28.6243684Z tokenizer_config.json: 11.7kB [00:00, 44.7MB/s] 2025-10-10T01:41:28.7001098Z 2025-10-10T01:41:28.7126205Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:28.7126523Z vocab.json: 1.86MB [00:00, 148MB/s] 2025-10-10T01:41:28.7491954Z 2025-10-10T01:41:28.7546638Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:41:28.7546931Z merges.txt: 1.08MB [00:00, 199MB/s] 2025-10-10T01:41:28.7921495Z 2025-10-10T01:41:28.8165436Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:28.8165795Z tokenizer.json: 8.03MB [00:00, 330MB/s] 2025-10-10T01:41:28.8535768Z 2025-10-10T01:41:28.8536898Z added_tokens.json: 0% 0.00/925 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:30.1436470Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:30.1437099Z 2025-10-10T01:41:30.1437620Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:30.1438294Z 2025-10-10T01:41:30.1439015Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:30.1439753Z 2025-10-10T01:41:30.1440144Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:30.1440536Z 2025-10-10T01:41:30.1440838Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1441117Z 2025-10-10T01:41:30.1441596Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:30.1442048Z 2025-10-10T01:41:30.1442382Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:30.1442712Z 2025-10-10T01:41:30.1443367Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:30.1443822Z 2025-10-10T01:41:30.1444147Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:30.1444666Z 2025-10-10T01:41:30.1444979Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1445259Z 2025-10-10T01:41:30.1445778Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:30.1446246Z 2025-10-10T01:41:30.1446495Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] self._init_executor() 2025-10-10T01:41:30.1446765Z 2025-10-10T01:41:30.1447309Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:30.1447801Z 2025-10-10T01:41:30.1448323Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:30.1448781Z 2025-10-10T01:41:30.1449421Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:30.1450255Z 2025-10-10T01:41:30.1450698Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:30.1451293Z 2025-10-10T01:41:30.1451761Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1452073Z 2025-10-10T01:41:30.1452757Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:30.1453363Z 2025-10-10T01:41:30.1453683Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:30.1453971Z 2025-10-10T01:41:30.1454229Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1454513Z 2025-10-10T01:41:30.1455051Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:30.1455526Z 2025-10-10T01:41:30.1455821Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:30.1456223Z 2025-10-10T01:41:30.1456501Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1456769Z 2025-10-10T01:41:30.1457328Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:30.1457834Z 2025-10-10T01:41:30.1458139Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:30.1458446Z 2025-10-10T01:41:30.1458717Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1458994Z 2025-10-10T01:41:30.1459438Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:30.1459855Z 2025-10-10T01:41:30.1460284Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:30.1460617Z 2025-10-10T01:41:30.1460901Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1461191Z 2025-10-10T01:41:30.1461534Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:30.1461863Z 2025-10-10T01:41:30.1462212Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:30.1462554Z 2025-10-10T01:41:30.1462933Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:30.1463289Z 2025-10-10T01:41:30.1463629Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:30.1463965Z 2025-10-10T01:41:30.1464321Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:30.1464671Z 2025-10-10T01:41:30.1465037Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:30.1465443Z 2025-10-10T01:41:30.1465934Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:30.1466399Z 2025-10-10T01:41:30.1466755Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:30.1467088Z 2025-10-10T01:41:30.1467614Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:30.1468090Z 2025-10-10T01:41:30.1468462Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:30.1468822Z 2025-10-10T01:41:30.1469386Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:30.1469894Z 2025-10-10T01:41:30.1470195Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:30.1470540Z 2025-10-10T01:41:30.1471168Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:30.1471739Z 2025-10-10T01:41:30.1472101Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:30.1472450Z 2025-10-10T01:41:30.1472701Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1472965Z 2025-10-10T01:41:30.1473518Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:30.1474038Z 2025-10-10T01:41:30.1474332Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:30.1474624Z 2025-10-10T01:41:30.1474974Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1475247Z 2025-10-10T01:41:30.1475856Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:30.1476399Z 2025-10-10T01:41:30.1476683Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:30.1476964Z 2025-10-10T01:41:30.1477221Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1477489Z 2025-10-10T01:41:30.1478061Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:30.1478590Z 2025-10-10T01:41:30.1478906Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:30.1479339Z 2025-10-10T01:41:30.1479611Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1479887Z 2025-10-10T01:41:30.1480405Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:30.1480931Z 2025-10-10T01:41:30.1481217Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:30.1481497Z 2025-10-10T01:41:30.1481761Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1482025Z 2025-10-10T01:41:30.1482547Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:30.1483024Z 2025-10-10T01:41:30.1483330Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:30.1483637Z 2025-10-10T01:41:30.1483873Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:30.1484136Z 2025-10-10T01:41:30.1484637Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:30.1485093Z 2025-10-10T01:41:30.1485337Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] raise RuntimeError( 2025-10-10T01:41:30.1485647Z 2025-10-10T01:41:30.1486224Z (EngineCore_DP0 pid=3190) ERROR 10-10 01:41:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:30.1486936Z (EngineCore_DP0 pid=3190) Process EngineCore_DP0: 2025-10-10T01:41:30.1487342Z (EngineCore_DP0 pid=3190) Traceback (most recent call last): 2025-10-10T01:41:30.1487946Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:30.1488452Z (EngineCore_DP0 pid=3190) self.run() 2025-10-10T01:41:30.1488973Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:30.1489527Z (EngineCore_DP0 pid=3190) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:30.1490300Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:30.1490854Z (EngineCore_DP0 pid=3190) raise e 2025-10-10T01:41:30.1491440Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:30.1492070Z (EngineCore_DP0 pid=3190) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:30.1492508Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1493128Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:30.1493763Z (EngineCore_DP0 pid=3190) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:30.1494413Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:30.1495032Z (EngineCore_DP0 pid=3190) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:30.1495477Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1496333Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:30.1496929Z (EngineCore_DP0 pid=3190) self._init_executor() 2025-10-10T01:41:30.1497596Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:30.1498412Z (EngineCore_DP0 pid=3190) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:30.1499133Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:30.1499845Z (EngineCore_DP0 pid=3190) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:30.1500339Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1500985Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:30.1501560Z (EngineCore_DP0 pid=3190) return func(*args, **kwargs) 2025-10-10T01:41:30.1501954Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1502592Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:30.1503232Z (EngineCore_DP0 pid=3190) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:30.1503740Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1504406Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:30.1505061Z (EngineCore_DP0 pid=3190) module = importlib.import_module(module_name) 2025-10-10T01:41:30.1505504Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1506083Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:30.1506709Z (EngineCore_DP0 pid=3190) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:30.1507207Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1507706Z (EngineCore_DP0 pid=3190) File "", line 1387, in _gcd_import 2025-10-10T01:41:30.1508374Z (EngineCore_DP0 pid=3190) File "", line 1360, in _find_and_load 2025-10-10T01:41:30.1508971Z (EngineCore_DP0 pid=3190) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:30.1509545Z (EngineCore_DP0 pid=3190) File "", line 935, in _load_unlocked 2025-10-10T01:41:30.1510113Z (EngineCore_DP0 pid=3190) File "", line 999, in exec_module 2025-10-10T01:41:30.1510712Z (EngineCore_DP0 pid=3190) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:30.1511440Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:30.1512118Z (EngineCore_DP0 pid=3190) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:30.1512843Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:30.1513575Z (EngineCore_DP0 pid=3190) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:30.1514348Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:30.1515018Z (EngineCore_DP0 pid=3190) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:30.1515854Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:30.1516660Z (EngineCore_DP0 pid=3190) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:30.1517156Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1517837Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:30.1518523Z (EngineCore_DP0 pid=3190) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:30.1518957Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1519750Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:30.1520455Z (EngineCore_DP0 pid=3190) return _is_fa2_supported(device)[0] 2025-10-10T01:41:30.1520857Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1521633Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:30.1522358Z (EngineCore_DP0 pid=3190) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:30.1522811Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1523482Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:30.1524113Z (EngineCore_DP0 pid=3190) prop = get_device_properties(device) 2025-10-10T01:41:30.1524523Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:30.1525176Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:30.1525836Z (EngineCore_DP0 pid=3190) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:30.1526311Z (EngineCore_DP0 pid=3190) ^^^^^^^^^^^^ 2025-10-10T01:41:30.1526894Z (EngineCore_DP0 pid=3190) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:30.1527444Z (EngineCore_DP0 pid=3190) raise RuntimeError( 2025-10-10T01:41:30.1528126Z (EngineCore_DP0 pid=3190) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:30.5508921Z FAILED 2025-10-10T01:41:30.5641216Z models/test_initialization.py::test_can_initialize_large_subset[MistralForCausalLM] Fork a new process to run a test 3194 2025-10-10T01:41:30.5651743Z Fork a new process to run a test 0 2025-10-10T01:41:30.5921223Z INFO 10-10 01:41:30 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MistralForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mistralai/Mistral-7B-Instruct-v0.1'} 2025-10-10T01:41:30.6881772Z 2025-10-10T01:41:30.6883123Z config.json: 0% 0.00/571 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:31.7826829Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:31.7827277Z 2025-10-10T01:41:31.7827770Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:31.7828174Z 2025-10-10T01:41:31.7828855Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:31.7829665Z 2025-10-10T01:41:31.7830065Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:31.7830453Z 2025-10-10T01:41:31.7830802Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7831149Z 2025-10-10T01:41:31.7831770Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:31.7832328Z 2025-10-10T01:41:31.7832667Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:31.7833129Z 2025-10-10T01:41:31.7833621Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:31.7834203Z 2025-10-10T01:41:31.7834528Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:31.7834970Z 2025-10-10T01:41:31.7835265Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7835538Z 2025-10-10T01:41:31.7836046Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:31.7836543Z 2025-10-10T01:41:31.7836920Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] self._init_executor() 2025-10-10T01:41:31.7837359Z 2025-10-10T01:41:31.7838391Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:31.7839411Z 2025-10-10T01:41:31.7840010Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:31.7840572Z 2025-10-10T01:41:31.7841749Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:31.7842465Z 2025-10-10T01:41:31.7842828Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:31.7843167Z 2025-10-10T01:41:31.7843457Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7843744Z 2025-10-10T01:41:31.7844234Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:31.7844716Z 2025-10-10T01:41:31.7844986Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:31.7845270Z 2025-10-10T01:41:31.7845526Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7845788Z 2025-10-10T01:41:31.7846297Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:31.7846786Z 2025-10-10T01:41:31.7847073Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:31.7847362Z 2025-10-10T01:41:31.7847699Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7847969Z 2025-10-10T01:41:31.7848502Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:31.7848985Z 2025-10-10T01:41:31.7849291Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:31.7849605Z 2025-10-10T01:41:31.7849881Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7850152Z 2025-10-10T01:41:31.7850593Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:31.7851001Z 2025-10-10T01:41:31.7851343Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:31.7851668Z 2025-10-10T01:41:31.7851955Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7852288Z 2025-10-10T01:41:31.7852639Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:31.7852967Z 2025-10-10T01:41:31.7853313Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:31.7853651Z 2025-10-10T01:41:31.7854025Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:31.7854385Z 2025-10-10T01:41:31.7854723Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:31.7855056Z 2025-10-10T01:41:31.7855412Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:31.7855760Z 2025-10-10T01:41:31.7856216Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:31.7856576Z 2025-10-10T01:41:31.7857075Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:31.7857530Z 2025-10-10T01:41:31.7857891Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:31.7858228Z 2025-10-10T01:41:31.7858751Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:31.7859223Z 2025-10-10T01:41:31.7859591Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:31.7859951Z 2025-10-10T01:41:31.7860504Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:31.7861011Z 2025-10-10T01:41:31.7861298Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:31.7861590Z 2025-10-10T01:41:31.7862214Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:31.7862837Z 2025-10-10T01:41:31.7863204Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:31.7863569Z 2025-10-10T01:41:31.7863832Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7864091Z 2025-10-10T01:41:31.7864666Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:31.7865189Z 2025-10-10T01:41:31.7865489Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:31.7865779Z 2025-10-10T01:41:31.7866046Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7866318Z 2025-10-10T01:41:31.7866910Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:31.7867514Z 2025-10-10T01:41:31.7867797Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:31.7868083Z 2025-10-10T01:41:31.7868338Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7868603Z 2025-10-10T01:41:31.7869174Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:31.7869695Z 2025-10-10T01:41:31.7870016Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:31.7870323Z 2025-10-10T01:41:31.7870594Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7870875Z 2025-10-10T01:41:31.7871476Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:31.7871957Z 2025-10-10T01:41:31.7872242Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:31.7872539Z 2025-10-10T01:41:31.7872797Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7873063Z 2025-10-10T01:41:31.7873579Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:31.7874061Z 2025-10-10T01:41:31.7874365Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:31.7874670Z 2025-10-10T01:41:31.7874896Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:31.7875133Z 2025-10-10T01:41:31.7875633Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:31.7876077Z 2025-10-10T01:41:31.7876326Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] raise RuntimeError( 2025-10-10T01:41:31.7876580Z 2025-10-10T01:41:31.7877143Z (EngineCore_DP0 pid=3222) ERROR 10-10 01:41:31 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:31.7877909Z (EngineCore_DP0 pid=3222) Process EngineCore_DP0: 2025-10-10T01:41:31.7878301Z (EngineCore_DP0 pid=3222) Traceback (most recent call last): 2025-10-10T01:41:31.7878905Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:31.7879537Z (EngineCore_DP0 pid=3222) self.run() 2025-10-10T01:41:31.7880058Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:31.7880612Z (EngineCore_DP0 pid=3222) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:31.7881288Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:31.7881837Z (EngineCore_DP0 pid=3222) raise e 2025-10-10T01:41:31.7882421Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:31.7883052Z (EngineCore_DP0 pid=3222) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:31.7883550Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7884169Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:31.7884802Z (EngineCore_DP0 pid=3222) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:31.7885459Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:31.7886082Z (EngineCore_DP0 pid=3222) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:31.7886528Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7887169Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:31.7887827Z (EngineCore_DP0 pid=3222) self._init_executor() 2025-10-10T01:41:31.7888506Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:31.7889189Z (EngineCore_DP0 pid=3222) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:31.7889904Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:31.7890607Z (EngineCore_DP0 pid=3222) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:31.7891095Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7891728Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:31.7892315Z (EngineCore_DP0 pid=3222) return func(*args, **kwargs) 2025-10-10T01:41:31.7892693Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7893316Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:31.7893937Z (EngineCore_DP0 pid=3222) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:31.7894352Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7895070Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:31.7895734Z (EngineCore_DP0 pid=3222) module = importlib.import_module(module_name) 2025-10-10T01:41:31.7896419Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7897051Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:31.7897684Z (EngineCore_DP0 pid=3222) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:31.7898187Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7898683Z (EngineCore_DP0 pid=3222) File "", line 1387, in _gcd_import 2025-10-10T01:41:31.7899239Z (EngineCore_DP0 pid=3222) File "", line 1360, in _find_and_load 2025-10-10T01:41:31.7899823Z (EngineCore_DP0 pid=3222) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:31.7900396Z (EngineCore_DP0 pid=3222) File "", line 935, in _load_unlocked 2025-10-10T01:41:31.7901078Z (EngineCore_DP0 pid=3222) File "", line 999, in exec_module 2025-10-10T01:41:31.7901679Z (EngineCore_DP0 pid=3222) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:31.7902411Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:31.7903091Z (EngineCore_DP0 pid=3222) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:31.7903814Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:31.7904532Z (EngineCore_DP0 pid=3222) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:31.7905427Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:31.7906115Z (EngineCore_DP0 pid=3222) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:31.7906899Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:31.7907707Z (EngineCore_DP0 pid=3222) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:31.7908180Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7908879Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:31.7909558Z (EngineCore_DP0 pid=3222) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:31.7909991Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7910731Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:31.7911414Z (EngineCore_DP0 pid=3222) return _is_fa2_supported(device)[0] 2025-10-10T01:41:31.7911822Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7912528Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:31.7913331Z (EngineCore_DP0 pid=3222) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:31.7913782Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7914455Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:31.7915091Z (EngineCore_DP0 pid=3222) prop = get_device_properties(device) 2025-10-10T01:41:31.7915512Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:31.7916167Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:31.7916827Z (EngineCore_DP0 pid=3222) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:31.7917217Z (EngineCore_DP0 pid=3222) ^^^^^^^^^^^^ 2025-10-10T01:41:31.7917789Z (EngineCore_DP0 pid=3222) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:31.7918345Z (EngineCore_DP0 pid=3222) raise RuntimeError( 2025-10-10T01:41:31.7919164Z (EngineCore_DP0 pid=3222) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:32.1741757Z FAILED 2025-10-10T01:41:32.1867974Z models/test_initialization.py::test_can_initialize_large_subset[ArcticForCausalLM] Fork a new process to run a test 3226 2025-10-10T01:41:32.1879307Z Fork a new process to run a test 0 2025-10-10T01:41:32.2157955Z INFO 10-10 01:41:32 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ArcticForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Snowflake/snowflake-arctic-instruct'} 2025-10-10T01:41:32.3054970Z 2025-10-10T01:41:32.3057094Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:32.3057594Z config.json: 1.28kB [00:00, 5.95MB/s] 2025-10-10T01:41:32.3835809Z 2025-10-10T01:41:32.3838232Z configuration_arctic.py: 0.00B [00:00, ?B/s] 2025-10-10T01:41:32.3838833Z configuration_arctic.py: 9.46kB [00:00, 52.8MB/s] 2025-10-10T01:41:32.3933950Z A new version of the following files was downloaded from https://huggingface.co/Snowflake/snowflake-arctic-instruct: 2025-10-10T01:41:32.3934881Z - configuration_arctic.py 2025-10-10T01:41:32.3935495Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:41:39.1734387Z INFO 10-10 01:41:39 [model.py:551] Resolved architecture: ArcticForCausalLM 2025-10-10T01:41:39.1735093Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:41:39.1974830Z INFO 10-10 01:41:39 [model.py:1545] Using max model len 4096 2025-10-10T01:41:39.1976624Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:41:39.2543062Z INFO 10-10 01:41:39 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:41:39.2927248Z 2025-10-10T01:41:39.2929138Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:41:39.2929975Z tokenizer_config.json: 1.58kB [00:00, 9.20MB/s] 2025-10-10T01:41:39.3354015Z 2025-10-10T01:41:39.3355715Z tokenization_arctic.py: 0.00B [00:00, ?B/s] 2025-10-10T01:41:39.3356272Z tokenization_arctic.py: 1.80kB [00:00, 17.1MB/s] 2025-10-10T01:41:39.3377021Z A new version of the following files was downloaded from https://huggingface.co/Snowflake/snowflake-arctic-instruct: 2025-10-10T01:41:39.3377936Z - tokenization_arctic.py 2025-10-10T01:41:39.3378755Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:41:39.5124582Z 2025-10-10T01:41:39.6236978Z tokenizer.model: 0% 0.00/500k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:40.0542940Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:40.0543522Z 2025-10-10T01:41:40.0544095Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:40.0544615Z 2025-10-10T01:41:40.0545579Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:40.0546779Z 2025-10-10T01:41:40.0547359Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:40.0547912Z 2025-10-10T01:41:40.0548411Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0548934Z 2025-10-10T01:41:40.0549848Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:40.0550729Z 2025-10-10T01:41:40.0551395Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:40.0552028Z 2025-10-10T01:41:40.0553014Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:40.0553898Z 2025-10-10T01:41:40.0554569Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:40.0555171Z 2025-10-10T01:41:40.0555760Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0556505Z 2025-10-10T01:41:40.0557532Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:40.0558424Z 2025-10-10T01:41:40.0558917Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] self._init_executor() 2025-10-10T01:41:40.0559541Z 2025-10-10T01:41:40.0560587Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:40.0561478Z 2025-10-10T01:41:40.0562020Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:40.0562554Z 2025-10-10T01:41:40.0563787Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:40.0564558Z 2025-10-10T01:41:40.0564952Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:40.0565305Z 2025-10-10T01:41:40.0565620Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0565900Z 2025-10-10T01:41:40.0566411Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:40.0566872Z 2025-10-10T01:41:40.0567143Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:40.0567411Z 2025-10-10T01:41:40.0567657Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0567924Z 2025-10-10T01:41:40.0568440Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:40.0568917Z 2025-10-10T01:41:40.0569202Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:40.0569503Z 2025-10-10T01:41:40.0569765Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0570682Z 2025-10-10T01:41:40.0571218Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:40.0571708Z 2025-10-10T01:41:40.0572014Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:40.0572317Z 2025-10-10T01:41:40.0572588Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0572861Z 2025-10-10T01:41:40.0573300Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:40.0573726Z 2025-10-10T01:41:40.0574064Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:40.0574391Z 2025-10-10T01:41:40.0574680Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0574954Z 2025-10-10T01:41:40.0575356Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:40.0575689Z 2025-10-10T01:41:40.0576037Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:40.0576367Z 2025-10-10T01:41:40.0576742Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:40.0577105Z 2025-10-10T01:41:40.0577446Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:40.0577782Z 2025-10-10T01:41:40.0578139Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:40.0578488Z 2025-10-10T01:41:40.0578976Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:40.0579344Z 2025-10-10T01:41:40.0579859Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:40.0580315Z 2025-10-10T01:41:40.0580680Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:40.0581024Z 2025-10-10T01:41:40.0581555Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:40.0582037Z 2025-10-10T01:41:40.0582411Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:40.0582779Z 2025-10-10T01:41:40.0583341Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:40.0583843Z 2025-10-10T01:41:40.0584133Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:40.0584436Z 2025-10-10T01:41:40.0585061Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:40.0585709Z 2025-10-10T01:41:40.0586075Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:40.0586440Z 2025-10-10T01:41:40.0586701Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0586964Z 2025-10-10T01:41:40.0587527Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:40.0588039Z 2025-10-10T01:41:40.0588342Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:40.0588632Z 2025-10-10T01:41:40.0588908Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0589176Z 2025-10-10T01:41:40.0589781Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:40.0590367Z 2025-10-10T01:41:40.0590660Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:40.0590949Z 2025-10-10T01:41:40.0591209Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0591479Z 2025-10-10T01:41:40.0592054Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:40.0592583Z 2025-10-10T01:41:40.0592898Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:40.0593215Z 2025-10-10T01:41:40.0593486Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0593755Z 2025-10-10T01:41:40.0594360Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:40.0594841Z 2025-10-10T01:41:40.0595134Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:40.0595419Z 2025-10-10T01:41:40.0595683Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0595945Z 2025-10-10T01:41:40.0596907Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:40.0597403Z 2025-10-10T01:41:40.0597724Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:40.0598032Z 2025-10-10T01:41:40.0598265Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:40.0598514Z 2025-10-10T01:41:40.0598994Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:40.0599554Z 2025-10-10T01:41:40.0599801Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] raise RuntimeError( 2025-10-10T01:41:40.0600054Z 2025-10-10T01:41:40.0600622Z (EngineCore_DP0 pid=3304) ERROR 10-10 01:41:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:40.0601417Z (EngineCore_DP0 pid=3304) Process EngineCore_DP0: 2025-10-10T01:41:40.0601802Z (EngineCore_DP0 pid=3304) Traceback (most recent call last): 2025-10-10T01:41:40.0602403Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:40.0602947Z (EngineCore_DP0 pid=3304) self.run() 2025-10-10T01:41:40.0603480Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:40.0604041Z (EngineCore_DP0 pid=3304) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:40.0604700Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:40.0605245Z (EngineCore_DP0 pid=3304) raise e 2025-10-10T01:41:40.0605827Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:40.0606457Z (EngineCore_DP0 pid=3304) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:40.0606966Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0607577Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:40.0608215Z (EngineCore_DP0 pid=3304) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:40.0608878Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:40.0609497Z (EngineCore_DP0 pid=3304) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:40.0609950Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0610586Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:40.0611170Z (EngineCore_DP0 pid=3304) self._init_executor() 2025-10-10T01:41:40.0611949Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:40.0612640Z (EngineCore_DP0 pid=3304) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:40.0613365Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:40.0614067Z (EngineCore_DP0 pid=3304) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:40.0614571Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0615206Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:40.0615794Z (EngineCore_DP0 pid=3304) return func(*args, **kwargs) 2025-10-10T01:41:40.0616182Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0616815Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:40.0617433Z (EngineCore_DP0 pid=3304) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:40.0617855Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0618511Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:40.0619210Z (EngineCore_DP0 pid=3304) module = importlib.import_module(module_name) 2025-10-10T01:41:40.0619648Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0620251Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:40.0620869Z (EngineCore_DP0 pid=3304) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:40.0621355Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0621847Z (EngineCore_DP0 pid=3304) File "", line 1387, in _gcd_import 2025-10-10T01:41:40.0622398Z (EngineCore_DP0 pid=3304) File "", line 1360, in _find_and_load 2025-10-10T01:41:40.0622992Z (EngineCore_DP0 pid=3304) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:40.0623579Z (EngineCore_DP0 pid=3304) File "", line 935, in _load_unlocked 2025-10-10T01:41:40.0624193Z (EngineCore_DP0 pid=3304) File "", line 999, in exec_module 2025-10-10T01:41:40.0624794Z (EngineCore_DP0 pid=3304) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:40.0625541Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:40.0626219Z (EngineCore_DP0 pid=3304) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:40.0626933Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:40.0627647Z (EngineCore_DP0 pid=3304) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:40.0628413Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:40.0629153Z (EngineCore_DP0 pid=3304) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:40.0629946Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:40.0630752Z (EngineCore_DP0 pid=3304) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:40.0631225Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0631911Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:40.0632599Z (EngineCore_DP0 pid=3304) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:40.0633032Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0633773Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:40.0634464Z (EngineCore_DP0 pid=3304) return _is_fa2_supported(device)[0] 2025-10-10T01:41:40.0634871Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0635566Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:40.0636319Z (EngineCore_DP0 pid=3304) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:40.0636767Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0637430Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:40.0638069Z (EngineCore_DP0 pid=3304) prop = get_device_properties(device) 2025-10-10T01:41:40.0638480Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:40.0639219Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:40.0639893Z (EngineCore_DP0 pid=3304) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:40.0640290Z (EngineCore_DP0 pid=3304) ^^^^^^^^^^^^ 2025-10-10T01:41:40.0640867Z (EngineCore_DP0 pid=3304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:40.0641418Z (EngineCore_DP0 pid=3304) raise RuntimeError( 2025-10-10T01:41:40.0642154Z (EngineCore_DP0 pid=3304) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:40.4461614Z FAILED 2025-10-10T01:41:40.4587874Z models/test_initialization.py::test_can_initialize_large_subset[StableLmForCausalLM] Fork a new process to run a test 3308 2025-10-10T01:41:40.4598820Z Fork a new process to run a test 0 2025-10-10T01:41:40.4870358Z INFO 10-10 01:41:40 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='StableLmForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'stabilityai/stablelm-3b-4e1t'} 2025-10-10T01:41:40.6421685Z 2025-10-10T01:41:40.6422862Z config.json: 0% 0.00/599 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:41:48.2475737Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] EngineCore failed to start. 2025-10-10T01:41:48.2476359Z 2025-10-10T01:41:48.2476983Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] Traceback (most recent call last): 2025-10-10T01:41:48.2477543Z 2025-10-10T01:41:48.2478329Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:48.2478824Z 2025-10-10T01:41:48.2479506Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:48.2479839Z 2025-10-10T01:41:48.2480137Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2480422Z 2025-10-10T01:41:48.2480912Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:48.2481358Z 2025-10-10T01:41:48.2481696Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:48.2482035Z 2025-10-10T01:41:48.2482521Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:48.2482966Z 2025-10-10T01:41:48.2483378Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:48.2483934Z 2025-10-10T01:41:48.2484445Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2484785Z 2025-10-10T01:41:48.2485314Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:48.2485785Z 2025-10-10T01:41:48.2486150Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] self._init_executor() 2025-10-10T01:41:48.2486428Z 2025-10-10T01:41:48.2486976Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:48.2487791Z 2025-10-10T01:41:48.2488400Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:48.2488956Z 2025-10-10T01:41:48.2489939Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:48.2490821Z 2025-10-10T01:41:48.2491423Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:48.2491923Z 2025-10-10T01:41:48.2492226Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2492525Z 2025-10-10T01:41:48.2493032Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:48.2493592Z 2025-10-10T01:41:48.2493902Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:41:48.2494177Z 2025-10-10T01:41:48.2494429Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2494684Z 2025-10-10T01:41:48.2495203Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:48.2495681Z 2025-10-10T01:41:48.2495978Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:48.2496487Z 2025-10-10T01:41:48.2496768Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2497051Z 2025-10-10T01:41:48.2497722Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:48.2498210Z 2025-10-10T01:41:48.2498517Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:41:48.2498824Z 2025-10-10T01:41:48.2499096Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2499375Z 2025-10-10T01:41:48.2499809Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:48.2500228Z 2025-10-10T01:41:48.2500579Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:48.2500914Z 2025-10-10T01:41:48.2501217Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2501499Z 2025-10-10T01:41:48.2501852Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:41:48.2502181Z 2025-10-10T01:41:48.2502532Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:41:48.2502866Z 2025-10-10T01:41:48.2503246Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:48.2503673Z 2025-10-10T01:41:48.2504014Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:41:48.2504352Z 2025-10-10T01:41:48.2504717Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:41:48.2505067Z 2025-10-10T01:41:48.2505440Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:48.2505801Z 2025-10-10T01:41:48.2506289Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:48.2506754Z 2025-10-10T01:41:48.2507110Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:48.2507440Z 2025-10-10T01:41:48.2507957Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:48.2508503Z 2025-10-10T01:41:48.2508894Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:48.2509252Z 2025-10-10T01:41:48.2509821Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:48.2510325Z 2025-10-10T01:41:48.2510616Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:41:48.2510918Z 2025-10-10T01:41:48.2511537Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:48.2512107Z 2025-10-10T01:41:48.2512543Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:48.2512901Z 2025-10-10T01:41:48.2513157Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2513422Z 2025-10-10T01:41:48.2513980Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:48.2514492Z 2025-10-10T01:41:48.2514792Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:41:48.2515085Z 2025-10-10T01:41:48.2515362Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2515634Z 2025-10-10T01:41:48.2516236Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:48.2516783Z 2025-10-10T01:41:48.2517068Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:41:48.2517349Z 2025-10-10T01:41:48.2517605Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2517874Z 2025-10-10T01:41:48.2518447Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:48.2519109Z 2025-10-10T01:41:48.2519433Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:48.2519749Z 2025-10-10T01:41:48.2520024Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2520295Z 2025-10-10T01:41:48.2520819Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:48.2521297Z 2025-10-10T01:41:48.2521578Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:41:48.2521860Z 2025-10-10T01:41:48.2522125Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2522392Z 2025-10-10T01:41:48.2522928Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:48.2523488Z 2025-10-10T01:41:48.2523806Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:41:48.2524115Z 2025-10-10T01:41:48.2524341Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:41:48.2524590Z 2025-10-10T01:41:48.2525075Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:48.2525536Z 2025-10-10T01:41:48.2525779Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] raise RuntimeError( 2025-10-10T01:41:48.2526040Z 2025-10-10T01:41:48.2526610Z (EngineCore_DP0 pid=3366) ERROR 10-10 01:41:48 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:48.2527305Z (EngineCore_DP0 pid=3366) Process EngineCore_DP0: 2025-10-10T01:41:48.2527805Z (EngineCore_DP0 pid=3366) Traceback (most recent call last): 2025-10-10T01:41:48.2528422Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:41:48.2528939Z (EngineCore_DP0 pid=3366) self.run() 2025-10-10T01:41:48.2529485Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:41:48.2530057Z (EngineCore_DP0 pid=3366) self._target(*self._args, **self._kwargs) 2025-10-10T01:41:48.2530724Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:41:48.2531275Z (EngineCore_DP0 pid=3366) raise e 2025-10-10T01:41:48.2531872Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:41:48.2532509Z (EngineCore_DP0 pid=3366) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:41:48.2532956Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2533575Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:41:48.2534213Z (EngineCore_DP0 pid=3366) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:41:48.2534925Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:41:48.2535540Z (EngineCore_DP0 pid=3366) self.model_executor = executor_class(vllm_config) 2025-10-10T01:41:48.2536011Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2536672Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:41:48.2537250Z (EngineCore_DP0 pid=3366) self._init_executor() 2025-10-10T01:41:48.2537915Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:41:48.2538598Z (EngineCore_DP0 pid=3366) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:41:48.2539321Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:41:48.2540034Z (EngineCore_DP0 pid=3366) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:41:48.2540587Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2541232Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:41:48.2541818Z (EngineCore_DP0 pid=3366) return func(*args, **kwargs) 2025-10-10T01:41:48.2542216Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2542853Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:41:48.2543479Z (EngineCore_DP0 pid=3366) worker_class = resolve_obj_by_qualname( 2025-10-10T01:41:48.2543905Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2544568Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:41:48.2545299Z (EngineCore_DP0 pid=3366) module = importlib.import_module(module_name) 2025-10-10T01:41:48.2545761Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2546339Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:41:48.2546982Z (EngineCore_DP0 pid=3366) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:41:48.2547485Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2547986Z (EngineCore_DP0 pid=3366) File "", line 1387, in _gcd_import 2025-10-10T01:41:48.2548544Z (EngineCore_DP0 pid=3366) File "", line 1360, in _find_and_load 2025-10-10T01:41:48.2549140Z (EngineCore_DP0 pid=3366) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:41:48.2549730Z (EngineCore_DP0 pid=3366) File "", line 935, in _load_unlocked 2025-10-10T01:41:48.2550303Z (EngineCore_DP0 pid=3366) File "", line 999, in exec_module 2025-10-10T01:41:48.2550908Z (EngineCore_DP0 pid=3366) File "", line 488, in _call_with_frames_removed 2025-10-10T01:41:48.2551649Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:41:48.2552400Z (EngineCore_DP0 pid=3366) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:41:48.2553118Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:41:48.2553859Z (EngineCore_DP0 pid=3366) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:41:48.2554642Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:41:48.2561639Z (EngineCore_DP0 pid=3366) class FlashAttentionMetadataBuilder( 2025-10-10T01:41:48.2562520Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:41:48.2563372Z (EngineCore_DP0 pid=3366) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:41:48.2563871Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2564575Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:41:48.2565368Z (EngineCore_DP0 pid=3366) if not is_fa_version_supported(fa_version): 2025-10-10T01:41:48.2565815Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2566572Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:41:48.2567295Z (EngineCore_DP0 pid=3366) return _is_fa2_supported(device)[0] 2025-10-10T01:41:48.2567723Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2568445Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:41:48.2569167Z (EngineCore_DP0 pid=3366) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:41:48.2569702Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2570395Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:41:48.2571041Z (EngineCore_DP0 pid=3366) prop = get_device_properties(device) 2025-10-10T01:41:48.2571460Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:41:48.2572116Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:41:48.2572779Z (EngineCore_DP0 pid=3366) _lazy_init() # will define _get_device_properties 2025-10-10T01:41:48.2573181Z (EngineCore_DP0 pid=3366) ^^^^^^^^^^^^ 2025-10-10T01:41:48.2573778Z (EngineCore_DP0 pid=3366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:41:48.2574340Z (EngineCore_DP0 pid=3366) raise RuntimeError( 2025-10-10T01:41:48.2575038Z (EngineCore_DP0 pid=3366) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:41:48.6383868Z FAILED 2025-10-10T01:41:48.6510868Z models/test_initialization.py::test_can_initialize_large_subset[Emu3ForConditionalGeneration] Fork a new process to run a test 3370 2025-10-10T01:41:48.6521955Z Fork a new process to run a test 0 2025-10-10T01:41:48.6794803Z INFO 10-10 01:41:48 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Emu3ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/Emu3-Chat-hf'} 2025-10-10T01:41:49.3476559Z INFO 10-10 01:41:49 [model.py:551] Resolved architecture: Emu3ForConditionalGeneration 2025-10-10T01:41:49.3477280Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:41:49.3719499Z INFO 10-10 01:41:49 [model.py:1765] Downcasting torch.float32 to torch.bfloat16. 2025-10-10T01:41:49.3720275Z INFO 10-10 01:41:49 [model.py:1545] Using max model len 131072 2025-10-10T01:41:49.6105335Z INFO 10-10 01:41:49 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:42:09.8522063Z (EngineCore_DP0 pid=3377) INFO 10-10 01:42:09 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:42:09.8592297Z (EngineCore_DP0 pid=3377) INFO 10-10 01:42:09 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='BAAI/Emu3-Chat-hf', speculative_config=None, tokenizer='BAAI/Emu3-Chat-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=BAAI/Emu3-Chat-hf, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:42:09.9932780Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] EngineCore failed to start. 2025-10-10T01:42:09.9933263Z 2025-10-10T01:42:09.9933662Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] Traceback (most recent call last): 2025-10-10T01:42:09.9934021Z 2025-10-10T01:42:09.9934696Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:09.9935295Z 2025-10-10T01:42:09.9935678Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:09.9936054Z 2025-10-10T01:42:09.9936410Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9936982Z 2025-10-10T01:42:09.9937544Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:09.9938011Z 2025-10-10T01:42:09.9938355Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:09.9938670Z 2025-10-10T01:42:09.9939294Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:09.9939758Z 2025-10-10T01:42:09.9940108Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:09.9940427Z 2025-10-10T01:42:09.9940741Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9941025Z 2025-10-10T01:42:09.9941531Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:09.9942136Z 2025-10-10T01:42:09.9942398Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] self._init_executor() 2025-10-10T01:42:09.9942673Z 2025-10-10T01:42:09.9943245Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:09.9944164Z 2025-10-10T01:42:09.9944804Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:09.9945286Z 2025-10-10T01:42:09.9946124Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:09.9946761Z 2025-10-10T01:42:09.9947365Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:09.9947729Z 2025-10-10T01:42:09.9948239Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9948541Z 2025-10-10T01:42:09.9949047Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:09.9949508Z 2025-10-10T01:42:09.9949776Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:42:09.9950061Z 2025-10-10T01:42:09.9950312Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9950586Z 2025-10-10T01:42:09.9951108Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:09.9951595Z 2025-10-10T01:42:09.9951886Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:09.9952185Z 2025-10-10T01:42:09.9952454Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9952722Z 2025-10-10T01:42:09.9953253Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:09.9953793Z 2025-10-10T01:42:09.9954107Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:42:09.9954419Z 2025-10-10T01:42:09.9954702Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9954980Z 2025-10-10T01:42:09.9955425Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:09.9955845Z 2025-10-10T01:42:09.9956184Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:09.9956518Z 2025-10-10T01:42:09.9956801Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9957102Z 2025-10-10T01:42:09.9957510Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:42:09.9957940Z 2025-10-10T01:42:09.9958322Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:42:09.9958722Z 2025-10-10T01:42:09.9959245Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:09.9959605Z 2025-10-10T01:42:09.9959959Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:42:09.9960298Z 2025-10-10T01:42:09.9960669Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:42:09.9961021Z 2025-10-10T01:42:09.9961392Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:09.9961761Z 2025-10-10T01:42:09.9962258Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:09.9962725Z 2025-10-10T01:42:09.9963158Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:09.9963501Z 2025-10-10T01:42:09.9964019Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:09.9964502Z 2025-10-10T01:42:09.9964870Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:09.9965226Z 2025-10-10T01:42:09.9965788Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:09.9966297Z 2025-10-10T01:42:09.9966613Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:42:09.9966921Z 2025-10-10T01:42:09.9967558Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:09.9968177Z 2025-10-10T01:42:09.9968822Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:09.9969197Z 2025-10-10T01:42:09.9969549Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9969821Z 2025-10-10T01:42:09.9970400Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:09.9970931Z 2025-10-10T01:42:09.9971230Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:42:09.9971531Z 2025-10-10T01:42:09.9971804Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9972077Z 2025-10-10T01:42:09.9972680Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:09.9973230Z 2025-10-10T01:42:09.9973513Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:42:09.9973796Z 2025-10-10T01:42:09.9974057Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9974371Z 2025-10-10T01:42:09.9974967Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:09.9975502Z 2025-10-10T01:42:09.9975821Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:09.9976140Z 2025-10-10T01:42:09.9976408Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9976684Z 2025-10-10T01:42:09.9977220Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:09.9977706Z 2025-10-10T01:42:09.9977996Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:42:09.9978287Z 2025-10-10T01:42:09.9978659Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9978943Z 2025-10-10T01:42:09.9979481Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:09.9979964Z 2025-10-10T01:42:09.9980280Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:42:09.9980584Z 2025-10-10T01:42:09.9980813Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:42:09.9981060Z 2025-10-10T01:42:09.9981539Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:09.9981992Z 2025-10-10T01:42:09.9982240Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] raise RuntimeError( 2025-10-10T01:42:09.9982505Z 2025-10-10T01:42:09.9983069Z (EngineCore_DP0 pid=3377) ERROR 10-10 01:42:09 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:09.9983763Z (EngineCore_DP0 pid=3377) Process EngineCore_DP0: 2025-10-10T01:42:09.9984178Z (EngineCore_DP0 pid=3377) Traceback (most recent call last): 2025-10-10T01:42:09.9984854Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:42:09.9985366Z (EngineCore_DP0 pid=3377) self.run() 2025-10-10T01:42:09.9985882Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:42:09.9986448Z (EngineCore_DP0 pid=3377) self._target(*self._args, **self._kwargs) 2025-10-10T01:42:09.9987115Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:42:09.9987658Z (EngineCore_DP0 pid=3377) raise e 2025-10-10T01:42:09.9988263Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:09.9988890Z (EngineCore_DP0 pid=3377) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:09.9989332Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9989948Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:09.9990648Z (EngineCore_DP0 pid=3377) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:09.9991312Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:09.9991921Z (EngineCore_DP0 pid=3377) self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:09.9992385Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9993040Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:09.9993625Z (EngineCore_DP0 pid=3377) self._init_executor() 2025-10-10T01:42:09.9994300Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:09.9994979Z (EngineCore_DP0 pid=3377) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:09.9995778Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:09.9996786Z (EngineCore_DP0 pid=3377) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:09.9997286Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9997927Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:09.9998515Z (EngineCore_DP0 pid=3377) return func(*args, **kwargs) 2025-10-10T01:42:09.9998892Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:09.9999601Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:10.0000249Z (EngineCore_DP0 pid=3377) worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:10.0000676Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0001339Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:10.0001987Z (EngineCore_DP0 pid=3377) module = importlib.import_module(module_name) 2025-10-10T01:42:10.0002428Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0003130Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:10.0003758Z (EngineCore_DP0 pid=3377) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:10.0004256Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0004749Z (EngineCore_DP0 pid=3377) File "", line 1387, in _gcd_import 2025-10-10T01:42:10.0005302Z (EngineCore_DP0 pid=3377) File "", line 1360, in _find_and_load 2025-10-10T01:42:10.0005892Z (EngineCore_DP0 pid=3377) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:10.0006468Z (EngineCore_DP0 pid=3377) File "", line 935, in _load_unlocked 2025-10-10T01:42:10.0007045Z (EngineCore_DP0 pid=3377) File "", line 999, in exec_module 2025-10-10T01:42:10.0007641Z (EngineCore_DP0 pid=3377) File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:10.0008505Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:10.0009205Z (EngineCore_DP0 pid=3377) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:10.0009933Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:10.0010652Z (EngineCore_DP0 pid=3377) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:10.0011422Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:10.0012092Z (EngineCore_DP0 pid=3377) class FlashAttentionMetadataBuilder( 2025-10-10T01:42:10.0012882Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:10.0013802Z (EngineCore_DP0 pid=3377) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:10.0014286Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0014977Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:10.0015653Z (EngineCore_DP0 pid=3377) if not is_fa_version_supported(fa_version): 2025-10-10T01:42:10.0016091Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0016838Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:10.0017535Z (EngineCore_DP0 pid=3377) return _is_fa2_supported(device)[0] 2025-10-10T01:42:10.0017945Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0018660Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:10.0019374Z (EngineCore_DP0 pid=3377) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:10.0019823Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0020489Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:10.0021172Z (EngineCore_DP0 pid=3377) prop = get_device_properties(device) 2025-10-10T01:42:10.0021580Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:10.0022261Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:10.0022919Z (EngineCore_DP0 pid=3377) _lazy_init() # will define _get_device_properties 2025-10-10T01:42:10.0023309Z (EngineCore_DP0 pid=3377) ^^^^^^^^^^^^ 2025-10-10T01:42:10.0023886Z (EngineCore_DP0 pid=3377) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:10.0024439Z (EngineCore_DP0 pid=3377) raise RuntimeError( 2025-10-10T01:42:10.0025131Z (EngineCore_DP0 pid=3377) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:10.4155516Z FAILED 2025-10-10T01:42:10.4283587Z models/test_initialization.py::test_can_initialize_large_subset[Exaone4ForCausalLM] Fork a new process to run a test 3381 2025-10-10T01:42:10.4293845Z Fork a new process to run a test 0 2025-10-10T01:42:10.4566843Z INFO 10-10 01:42:10 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Exaone4ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'LGAI-EXAONE/EXAONE-4.0-32B'} 2025-10-10T01:42:10.6479787Z 2025-10-10T01:42:10.6481963Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:10.6482284Z config.json: 2.42kB [00:00, 11.5MB/s] 2025-10-10T01:42:17.4292959Z INFO 10-10 01:42:17 [model.py:551] Resolved architecture: Exaone4ForCausalLM 2025-10-10T01:42:17.4293461Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:42:17.4533869Z INFO 10-10 01:42:17 [model.py:1545] Using max model len 131072 2025-10-10T01:42:17.6252271Z INFO 10-10 01:42:17 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:42:17.6634996Z 2025-10-10T01:42:17.6639760Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:17.6640171Z tokenizer_config.json: 70.3kB [00:00, 120MB/s] 2025-10-10T01:42:17.7378005Z 2025-10-10T01:42:17.7499155Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:17.7499446Z vocab.json: 1.93MB [00:00, 160MB/s] 2025-10-10T01:42:17.7927201Z 2025-10-10T01:42:17.7988703Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:42:17.7988990Z merges.txt: 1.22MB [00:00, 200MB/s] 2025-10-10T01:42:17.8488647Z 2025-10-10T01:42:17.8732243Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:17.8732592Z tokenizer.json: 7.91MB [00:00, 325MB/s] 2025-10-10T01:42:17.9587224Z 2025-10-10T01:42:17.9588870Z special_tokens_map.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:17.9589264Z special_tokens_map.json: 6.70kB [00:00, 55.6MB/s] 2025-10-10T01:42:17.9968195Z 2025-10-10T01:42:17.9969772Z chat_template.jinja: 0.00B [00:00, ?B/s] 2025-10-10T01:42:17.9970122Z chat_template.jinja: 5.49kB [00:00, 45.5MB/s] 2025-10-10T01:42:18.3076497Z 2025-10-10T01:42:18.3077366Z generation_config.json: 0% 0.00/170 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:42:18.4732868Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] EngineCore failed to start. 2025-10-10T01:42:18.4733306Z 2025-10-10T01:42:18.4733708Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] Traceback (most recent call last): 2025-10-10T01:42:18.4734079Z 2025-10-10T01:42:18.4734761Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:18.4735281Z 2025-10-10T01:42:18.4735901Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:18.4736226Z 2025-10-10T01:42:18.4736521Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4736799Z 2025-10-10T01:42:18.4737291Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:18.4737751Z 2025-10-10T01:42:18.4738110Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:18.4738589Z 2025-10-10T01:42:18.4739082Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:18.4739551Z 2025-10-10T01:42:18.4739865Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:18.4740178Z 2025-10-10T01:42:18.4740452Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4740740Z 2025-10-10T01:42:18.4741257Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:18.4741843Z 2025-10-10T01:42:18.4742105Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] self._init_executor() 2025-10-10T01:42:18.4742474Z 2025-10-10T01:42:18.4743364Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:18.4744312Z 2025-10-10T01:42:18.4744823Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:18.4745313Z 2025-10-10T01:42:18.4745888Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:18.4746389Z 2025-10-10T01:42:18.4746733Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:18.4747078Z 2025-10-10T01:42:18.4747364Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4747646Z 2025-10-10T01:42:18.4748279Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:18.4748734Z 2025-10-10T01:42:18.4748999Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:42:18.4749265Z 2025-10-10T01:42:18.4749518Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4749772Z 2025-10-10T01:42:18.4750289Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:18.4750762Z 2025-10-10T01:42:18.4751055Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:18.4751340Z 2025-10-10T01:42:18.4751605Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4751888Z 2025-10-10T01:42:18.4752495Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:18.4752994Z 2025-10-10T01:42:18.4753299Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:42:18.4753599Z 2025-10-10T01:42:18.4753869Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4754147Z 2025-10-10T01:42:18.4754584Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:18.4755001Z 2025-10-10T01:42:18.4755342Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:18.4755671Z 2025-10-10T01:42:18.4755957Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4756230Z 2025-10-10T01:42:18.4756573Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:42:18.4756898Z 2025-10-10T01:42:18.4757250Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:42:18.4757643Z 2025-10-10T01:42:18.4758030Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:18.4758398Z 2025-10-10T01:42:18.4758746Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:42:18.4759205Z 2025-10-10T01:42:18.4759573Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:42:18.4759942Z 2025-10-10T01:42:18.4760318Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:18.4760681Z 2025-10-10T01:42:18.4761185Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:18.4761655Z 2025-10-10T01:42:18.4762020Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:18.4762422Z 2025-10-10T01:42:18.4762962Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:18.4763439Z 2025-10-10T01:42:18.4763819Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:18.4764177Z 2025-10-10T01:42:18.4764725Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:18.4765233Z 2025-10-10T01:42:18.4765523Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:42:18.4765822Z 2025-10-10T01:42:18.4766449Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:18.4767028Z 2025-10-10T01:42:18.4767468Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:18.4767831Z 2025-10-10T01:42:18.4768100Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4768362Z 2025-10-10T01:42:18.4768923Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:18.4769438Z 2025-10-10T01:42:18.4769734Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:42:18.4770025Z 2025-10-10T01:42:18.4770301Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4770574Z 2025-10-10T01:42:18.4771171Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:18.4771721Z 2025-10-10T01:42:18.4771999Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:42:18.4772296Z 2025-10-10T01:42:18.4772556Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4772918Z 2025-10-10T01:42:18.4773494Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:18.4774022Z 2025-10-10T01:42:18.4774397Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:18.4774768Z 2025-10-10T01:42:18.4775104Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4775429Z 2025-10-10T01:42:18.4776045Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:18.4776608Z 2025-10-10T01:42:18.4776888Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:42:18.4777170Z 2025-10-10T01:42:18.4777429Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4777698Z 2025-10-10T01:42:18.4778227Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:18.4778794Z 2025-10-10T01:42:18.4779114Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:42:18.4779418Z 2025-10-10T01:42:18.4779639Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:42:18.4779884Z 2025-10-10T01:42:18.4780359Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:18.4780808Z 2025-10-10T01:42:18.4781059Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] raise RuntimeError( 2025-10-10T01:42:18.4781312Z 2025-10-10T01:42:18.4781882Z (EngineCore_DP0 pid=3439) ERROR 10-10 01:42:18 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:18.4782649Z (EngineCore_DP0 pid=3439) Process EngineCore_DP0: 2025-10-10T01:42:18.4783082Z (EngineCore_DP0 pid=3439) Traceback (most recent call last): 2025-10-10T01:42:18.4783691Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:42:18.4784238Z (EngineCore_DP0 pid=3439) self.run() 2025-10-10T01:42:18.4784859Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:42:18.4785556Z (EngineCore_DP0 pid=3439) self._target(*self._args, **self._kwargs) 2025-10-10T01:42:18.4786316Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:42:18.4786869Z (EngineCore_DP0 pid=3439) raise e 2025-10-10T01:42:18.4787474Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:18.4788101Z (EngineCore_DP0 pid=3439) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:18.4788545Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4789174Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:18.4789873Z (EngineCore_DP0 pid=3439) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:18.4790527Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:18.4791145Z (EngineCore_DP0 pid=3439) self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:18.4791604Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4792257Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:18.4792834Z (EngineCore_DP0 pid=3439) self._init_executor() 2025-10-10T01:42:18.4793493Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:18.4794171Z (EngineCore_DP0 pid=3439) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:18.4794898Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:18.4795666Z (EngineCore_DP0 pid=3439) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:18.4796414Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4797069Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:18.4797651Z (EngineCore_DP0 pid=3439) return func(*args, **kwargs) 2025-10-10T01:42:18.4798031Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4798664Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:18.4799360Z (EngineCore_DP0 pid=3439) worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:18.4799782Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4800603Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:18.4801260Z (EngineCore_DP0 pid=3439) module = importlib.import_module(module_name) 2025-10-10T01:42:18.4801712Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4802292Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:18.4802913Z (EngineCore_DP0 pid=3439) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:18.4803406Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4803893Z (EngineCore_DP0 pid=3439) File "", line 1387, in _gcd_import 2025-10-10T01:42:18.4804448Z (EngineCore_DP0 pid=3439) File "", line 1360, in _find_and_load 2025-10-10T01:42:18.4805036Z (EngineCore_DP0 pid=3439) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:18.4805613Z (EngineCore_DP0 pid=3439) File "", line 935, in _load_unlocked 2025-10-10T01:42:18.4806183Z (EngineCore_DP0 pid=3439) File "", line 999, in exec_module 2025-10-10T01:42:18.4806779Z (EngineCore_DP0 pid=3439) File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:18.4807592Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:18.4808266Z (EngineCore_DP0 pid=3439) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:18.4808987Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:18.4809708Z (EngineCore_DP0 pid=3439) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:18.4810474Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:18.4811141Z (EngineCore_DP0 pid=3439) class FlashAttentionMetadataBuilder( 2025-10-10T01:42:18.4811930Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:18.4812740Z (EngineCore_DP0 pid=3439) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:18.4813221Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4813999Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:18.4814675Z (EngineCore_DP0 pid=3439) if not is_fa_version_supported(fa_version): 2025-10-10T01:42:18.4815112Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4815853Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:18.4816548Z (EngineCore_DP0 pid=3439) return _is_fa2_supported(device)[0] 2025-10-10T01:42:18.4816960Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4817679Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:18.4818495Z (EngineCore_DP0 pid=3439) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:18.4818962Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4819634Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:18.4820266Z (EngineCore_DP0 pid=3439) prop = get_device_properties(device) 2025-10-10T01:42:18.4820674Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:18.4821336Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:18.4822008Z (EngineCore_DP0 pid=3439) _lazy_init() # will define _get_device_properties 2025-10-10T01:42:18.4822410Z (EngineCore_DP0 pid=3439) ^^^^^^^^^^^^ 2025-10-10T01:42:18.4822990Z (EngineCore_DP0 pid=3439) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:18.4823542Z (EngineCore_DP0 pid=3439) raise RuntimeError( 2025-10-10T01:42:18.4824225Z (EngineCore_DP0 pid=3439) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:18.8768753Z FAILED 2025-10-10T01:42:18.8895317Z models/test_initialization.py::test_can_initialize_large_subset[Glm4ForCausalLM] Fork a new process to run a test 3443 2025-10-10T01:42:18.8905451Z Fork a new process to run a test 0 2025-10-10T01:42:18.9177783Z INFO 10-10 01:42:18 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Glm4ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'zai-org/GLM-4-9B-0414'} 2025-10-10T01:42:19.0878940Z 2025-10-10T01:42:19.0880345Z config.json: 0% 0.00/689 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:42:27.5384353Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] EngineCore failed to start. 2025-10-10T01:42:27.5384987Z 2025-10-10T01:42:27.5385408Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] Traceback (most recent call last): 2025-10-10T01:42:27.5385811Z 2025-10-10T01:42:27.5386482Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:27.5387071Z 2025-10-10T01:42:27.5387466Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:27.5387872Z 2025-10-10T01:42:27.5388225Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5388566Z 2025-10-10T01:42:27.5389473Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:27.5390194Z 2025-10-10T01:42:27.5390860Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:27.5391286Z 2025-10-10T01:42:27.5391952Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:27.5392531Z 2025-10-10T01:42:27.5392924Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:27.5393284Z 2025-10-10T01:42:27.5393574Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5394076Z 2025-10-10T01:42:27.5395301Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:27.5396061Z 2025-10-10T01:42:27.5396675Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] self._init_executor() 2025-10-10T01:42:27.5397059Z 2025-10-10T01:42:27.5398024Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:27.5398579Z 2025-10-10T01:42:27.5398960Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:27.5399401Z 2025-10-10T01:42:27.5399951Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:27.5400460Z 2025-10-10T01:42:27.5400814Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:27.5401168Z 2025-10-10T01:42:27.5401462Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5401748Z 2025-10-10T01:42:27.5402246Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:27.5402865Z 2025-10-10T01:42:27.5403143Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:42:27.5403416Z 2025-10-10T01:42:27.5403680Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5403941Z 2025-10-10T01:42:27.5404466Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:27.5404946Z 2025-10-10T01:42:27.5405242Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:27.5405531Z 2025-10-10T01:42:27.5405795Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5406069Z 2025-10-10T01:42:27.5406595Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:27.5407084Z 2025-10-10T01:42:27.5407383Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:42:27.5407759Z 2025-10-10T01:42:27.5408035Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5408314Z 2025-10-10T01:42:27.5408754Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:27.5409158Z 2025-10-10T01:42:27.5409499Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:27.5409823Z 2025-10-10T01:42:27.5410116Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5410390Z 2025-10-10T01:42:27.5410731Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:42:27.5411072Z 2025-10-10T01:42:27.5411537Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:42:27.5411886Z 2025-10-10T01:42:27.5412258Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:27.5412638Z 2025-10-10T01:42:27.5412987Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:42:27.5413323Z 2025-10-10T01:42:27.5413684Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:42:27.5414038Z 2025-10-10T01:42:27.5414408Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:27.5414768Z 2025-10-10T01:42:27.5415267Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:27.5415727Z 2025-10-10T01:42:27.5416081Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:27.5416414Z 2025-10-10T01:42:27.5416930Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:27.5417463Z 2025-10-10T01:42:27.5417834Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:27.5418195Z 2025-10-10T01:42:27.5418744Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:27.5419254Z 2025-10-10T01:42:27.5419548Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:42:27.5419846Z 2025-10-10T01:42:27.5420468Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:27.5421044Z 2025-10-10T01:42:27.5421405Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:27.5421758Z 2025-10-10T01:42:27.5422009Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5422310Z 2025-10-10T01:42:27.5422882Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:27.5423395Z 2025-10-10T01:42:27.5423707Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:42:27.5424000Z 2025-10-10T01:42:27.5424276Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5424546Z 2025-10-10T01:42:27.5425139Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:27.5425690Z 2025-10-10T01:42:27.5425967Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:42:27.5426254Z 2025-10-10T01:42:27.5426619Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5426897Z 2025-10-10T01:42:27.5427475Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:27.5428004Z 2025-10-10T01:42:27.5428321Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:27.5428627Z 2025-10-10T01:42:27.5428903Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5429172Z 2025-10-10T01:42:27.5429701Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:27.5430181Z 2025-10-10T01:42:27.5430464Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:42:27.5430746Z 2025-10-10T01:42:27.5431007Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5431277Z 2025-10-10T01:42:27.5431794Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:27.5432294Z 2025-10-10T01:42:27.5432649Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:42:27.5432967Z 2025-10-10T01:42:27.5433191Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:42:27.5433434Z 2025-10-10T01:42:27.5434074Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:27.5434526Z 2025-10-10T01:42:27.5434779Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] raise RuntimeError( 2025-10-10T01:42:27.5435033Z 2025-10-10T01:42:27.5435606Z (EngineCore_DP0 pid=3521) ERROR 10-10 01:42:27 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:27.5436292Z (EngineCore_DP0 pid=3521) Process EngineCore_DP0: 2025-10-10T01:42:27.5436717Z (EngineCore_DP0 pid=3521) Traceback (most recent call last): 2025-10-10T01:42:27.5437322Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:42:27.5437900Z (EngineCore_DP0 pid=3521) self.run() 2025-10-10T01:42:27.5438428Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:42:27.5438997Z (EngineCore_DP0 pid=3521) self._target(*self._args, **self._kwargs) 2025-10-10T01:42:27.5439757Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:42:27.5440300Z (EngineCore_DP0 pid=3521) raise e 2025-10-10T01:42:27.5440909Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:27.5441546Z (EngineCore_DP0 pid=3521) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:27.5441987Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5442684Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:27.5443322Z (EngineCore_DP0 pid=3521) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:27.5443992Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:27.5444608Z (EngineCore_DP0 pid=3521) self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:27.5445060Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5445706Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:27.5446282Z (EngineCore_DP0 pid=3521) self._init_executor() 2025-10-10T01:42:27.5446944Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:27.5447626Z (EngineCore_DP0 pid=3521) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:27.5448364Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:27.5449070Z (EngineCore_DP0 pid=3521) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:27.5449562Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5450249Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:27.5450846Z (EngineCore_DP0 pid=3521) return func(*args, **kwargs) 2025-10-10T01:42:27.5451232Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5451872Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:27.5452507Z (EngineCore_DP0 pid=3521) worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:27.5452936Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5453604Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:27.5454261Z (EngineCore_DP0 pid=3521) module = importlib.import_module(module_name) 2025-10-10T01:42:27.5454702Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5455281Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:27.5455966Z (EngineCore_DP0 pid=3521) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:27.5456459Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5456952Z (EngineCore_DP0 pid=3521) File "", line 1387, in _gcd_import 2025-10-10T01:42:27.5457517Z (EngineCore_DP0 pid=3521) File "", line 1360, in _find_and_load 2025-10-10T01:42:27.5458113Z (EngineCore_DP0 pid=3521) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:27.5458885Z (EngineCore_DP0 pid=3521) File "", line 935, in _load_unlocked 2025-10-10T01:42:27.5459483Z (EngineCore_DP0 pid=3521) File "", line 999, in exec_module 2025-10-10T01:42:27.5460181Z (EngineCore_DP0 pid=3521) File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:27.5460944Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:27.5461629Z (EngineCore_DP0 pid=3521) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:27.5462347Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:27.5463076Z (EngineCore_DP0 pid=3521) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:27.5463855Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:27.5464524Z (EngineCore_DP0 pid=3521) class FlashAttentionMetadataBuilder( 2025-10-10T01:42:27.5465319Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:27.5466144Z (EngineCore_DP0 pid=3521) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:27.5466621Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5467312Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:27.5468042Z (EngineCore_DP0 pid=3521) if not is_fa_version_supported(fa_version): 2025-10-10T01:42:27.5468479Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5469220Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:27.5469914Z (EngineCore_DP0 pid=3521) return _is_fa2_supported(device)[0] 2025-10-10T01:42:27.5470326Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5471047Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:27.5471764Z (EngineCore_DP0 pid=3521) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:27.5472219Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5472890Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:27.5473577Z (EngineCore_DP0 pid=3521) prop = get_device_properties(device) 2025-10-10T01:42:27.5473997Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:27.5474650Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:27.5475315Z (EngineCore_DP0 pid=3521) _lazy_init() # will define _get_device_properties 2025-10-10T01:42:27.5475700Z (EngineCore_DP0 pid=3521) ^^^^^^^^^^^^ 2025-10-10T01:42:27.5476274Z (EngineCore_DP0 pid=3521) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:27.5476836Z (EngineCore_DP0 pid=3521) raise RuntimeError( 2025-10-10T01:42:27.5477516Z (EngineCore_DP0 pid=3521) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:27.9539605Z FAILED 2025-10-10T01:42:27.9667474Z models/test_initialization.py::test_can_initialize_large_subset[BailingMoeForCausalLM] Fork a new process to run a test 3525 2025-10-10T01:42:27.9677950Z Fork a new process to run a test 0 2025-10-10T01:42:27.9953786Z INFO 10-10 01:42:27 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BailingMoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'inclusionAI/Ling-lite-1.5'} 2025-10-10T01:42:28.1217700Z 2025-10-10T01:42:28.1219829Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:28.1220142Z config.json: 1.28kB [00:00, 7.25MB/s] 2025-10-10T01:42:28.2073757Z 2025-10-10T01:42:28.2075898Z configuration_bailing_moe.py: 0.00B [00:00, ?B/s] 2025-10-10T01:42:28.2076314Z configuration_bailing_moe.py: 2.86kB [00:00, 12.0MB/s] 2025-10-10T01:42:28.2164825Z A new version of the following files was downloaded from https://huggingface.co/inclusionAI/Ling-lite-1.5: 2025-10-10T01:42:28.2165382Z - configuration_bailing_moe.py 2025-10-10T01:42:28.2166022Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:42:35.0033039Z INFO 10-10 01:42:35 [model.py:551] Resolved architecture: BailingMoeForCausalLM 2025-10-10T01:42:35.0033707Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:42:35.0280297Z INFO 10-10 01:42:35 [model.py:1545] Using max model len 32768 2025-10-10T01:42:35.0281447Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:42:35.0895225Z INFO 10-10 01:42:35 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:42:35.1293316Z 2025-10-10T01:42:35.1296497Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:35.1296897Z tokenizer_config.json: 51.1kB [00:00, 176MB/s] 2025-10-10T01:42:35.2032131Z 2025-10-10T01:42:35.2364797Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:35.2365121Z tokenizer.json: 9.75MB [00:00, 294MB/s] 2025-10-10T01:42:35.3384911Z 2025-10-10T01:42:35.3385985Z special_tokens_map.json: 0% 0.00/747 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:42:36.0449986Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:42:36.0450498Z 2025-10-10T01:42:36.0451014Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:42:36.0451608Z 2025-10-10T01:42:36.0452494Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:36.0453113Z 2025-10-10T01:42:36.0453510Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:36.0453893Z 2025-10-10T01:42:36.0454238Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0454890Z 2025-10-10T01:42:36.0455502Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:36.0456062Z 2025-10-10T01:42:36.0456652Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:36.0457065Z 2025-10-10T01:42:36.0457863Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:36.0458433Z 2025-10-10T01:42:36.0458873Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:36.0459263Z 2025-10-10T01:42:36.0459630Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0460140Z 2025-10-10T01:42:36.0460891Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:36.0461518Z 2025-10-10T01:42:36.0462218Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] self._init_executor() 2025-10-10T01:42:36.0462660Z 2025-10-10T01:42:36.0463444Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:36.0464219Z 2025-10-10T01:42:36.0464584Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:36.0464924Z 2025-10-10T01:42:36.0465483Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:36.0466005Z 2025-10-10T01:42:36.0466356Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:36.0466691Z 2025-10-10T01:42:36.0467159Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0467444Z 2025-10-10T01:42:36.0467947Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:36.0468396Z 2025-10-10T01:42:36.0468658Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:42:36.0468933Z 2025-10-10T01:42:36.0469181Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0469459Z 2025-10-10T01:42:36.0469972Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:36.0470453Z 2025-10-10T01:42:36.0470749Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:36.0471041Z 2025-10-10T01:42:36.0471304Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0471571Z 2025-10-10T01:42:36.0472103Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:36.0472582Z 2025-10-10T01:42:36.0472894Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:42:36.0473235Z 2025-10-10T01:42:36.0473512Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0473782Z 2025-10-10T01:42:36.0474221Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:36.0474651Z 2025-10-10T01:42:36.0474990Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:36.0475330Z 2025-10-10T01:42:36.0475616Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0475900Z 2025-10-10T01:42:36.0476237Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:42:36.0476567Z 2025-10-10T01:42:36.0476917Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:42:36.0477251Z 2025-10-10T01:42:36.0477682Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:36.0478056Z 2025-10-10T01:42:36.0478402Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:42:36.0478733Z 2025-10-10T01:42:36.0479228Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:42:36.0479580Z 2025-10-10T01:42:36.0479952Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:36.0480319Z 2025-10-10T01:42:36.0480823Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:36.0481289Z 2025-10-10T01:42:36.0481715Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:36.0482057Z 2025-10-10T01:42:36.0482571Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:36.0483048Z 2025-10-10T01:42:36.0483416Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:36.0483768Z 2025-10-10T01:42:36.0484321Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:36.0484823Z 2025-10-10T01:42:36.0485117Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:42:36.0485410Z 2025-10-10T01:42:36.0486038Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:36.0486607Z 2025-10-10T01:42:36.0486974Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:36.0487321Z 2025-10-10T01:42:36.0487570Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0487877Z 2025-10-10T01:42:36.0488434Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:36.0488947Z 2025-10-10T01:42:36.0489240Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:42:36.0489541Z 2025-10-10T01:42:36.0489809Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0490076Z 2025-10-10T01:42:36.0490675Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:36.0491217Z 2025-10-10T01:42:36.0491497Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:42:36.0491777Z 2025-10-10T01:42:36.0492037Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0492305Z 2025-10-10T01:42:36.0492885Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:36.0493449Z 2025-10-10T01:42:36.0493763Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:36.0494077Z 2025-10-10T01:42:36.0494346Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0494632Z 2025-10-10T01:42:36.0495155Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:36.0495639Z 2025-10-10T01:42:36.0495913Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:42:36.0496414Z 2025-10-10T01:42:36.0496693Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0497114Z 2025-10-10T01:42:36.0497678Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:36.0498166Z 2025-10-10T01:42:36.0498479Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:42:36.0498778Z 2025-10-10T01:42:36.0499019Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:42:36.0499274Z 2025-10-10T01:42:36.0499756Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:36.0500206Z 2025-10-10T01:42:36.0500453Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] raise RuntimeError( 2025-10-10T01:42:36.0500713Z 2025-10-10T01:42:36.0501277Z (EngineCore_DP0 pid=3583) ERROR 10-10 01:42:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:36.0501969Z (EngineCore_DP0 pid=3583) Process EngineCore_DP0: 2025-10-10T01:42:36.0502362Z (EngineCore_DP0 pid=3583) Traceback (most recent call last): 2025-10-10T01:42:36.0502981Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:42:36.0503578Z (EngineCore_DP0 pid=3583) self.run() 2025-10-10T01:42:36.0504105Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:42:36.0504669Z (EngineCore_DP0 pid=3583) self._target(*self._args, **self._kwargs) 2025-10-10T01:42:36.0505343Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:42:36.0505893Z (EngineCore_DP0 pid=3583) raise e 2025-10-10T01:42:36.0506480Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:36.0507102Z (EngineCore_DP0 pid=3583) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:36.0507543Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0508177Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:36.0508817Z (EngineCore_DP0 pid=3583) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:36.0509548Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:36.0510154Z (EngineCore_DP0 pid=3583) self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:36.0510611Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0511252Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:36.0511829Z (EngineCore_DP0 pid=3583) self._init_executor() 2025-10-10T01:42:36.0512498Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:36.0513191Z (EngineCore_DP0 pid=3583) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:36.0513998Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:36.0514712Z (EngineCore_DP0 pid=3583) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:36.0515201Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0515837Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:36.0516432Z (EngineCore_DP0 pid=3583) return func(*args, **kwargs) 2025-10-10T01:42:36.0516814Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0517444Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:36.0518070Z (EngineCore_DP0 pid=3583) worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:36.0518495Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0519213Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:36.0519859Z (EngineCore_DP0 pid=3583) module = importlib.import_module(module_name) 2025-10-10T01:42:36.0520300Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0520930Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:36.0521562Z (EngineCore_DP0 pid=3583) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:36.0522055Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0522549Z (EngineCore_DP0 pid=3583) File "", line 1387, in _gcd_import 2025-10-10T01:42:36.0523100Z (EngineCore_DP0 pid=3583) File "", line 1360, in _find_and_load 2025-10-10T01:42:36.0523687Z (EngineCore_DP0 pid=3583) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:36.0524261Z (EngineCore_DP0 pid=3583) File "", line 935, in _load_unlocked 2025-10-10T01:42:36.0524828Z (EngineCore_DP0 pid=3583) File "", line 999, in exec_module 2025-10-10T01:42:36.0525425Z (EngineCore_DP0 pid=3583) File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:36.0526167Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:36.0526894Z (EngineCore_DP0 pid=3583) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:36.0527646Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:36.0528381Z (EngineCore_DP0 pid=3583) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:36.0529165Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:36.0529835Z (EngineCore_DP0 pid=3583) class FlashAttentionMetadataBuilder( 2025-10-10T01:42:36.0530639Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:36.0531527Z (EngineCore_DP0 pid=3583) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:36.0532011Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0532701Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:36.0533377Z (EngineCore_DP0 pid=3583) if not is_fa_version_supported(fa_version): 2025-10-10T01:42:36.0533817Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0534581Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:36.0535276Z (EngineCore_DP0 pid=3583) return _is_fa2_supported(device)[0] 2025-10-10T01:42:36.0535691Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0536396Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:36.0537113Z (EngineCore_DP0 pid=3583) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:36.0537568Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0538237Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:36.0538923Z (EngineCore_DP0 pid=3583) prop = get_device_properties(device) 2025-10-10T01:42:36.0539350Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:36.0540000Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:36.0540673Z (EngineCore_DP0 pid=3583) _lazy_init() # will define _get_device_properties 2025-10-10T01:42:36.0541064Z (EngineCore_DP0 pid=3583) ^^^^^^^^^^^^ 2025-10-10T01:42:36.0541645Z (EngineCore_DP0 pid=3583) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:36.0542198Z (EngineCore_DP0 pid=3583) raise RuntimeError( 2025-10-10T01:42:36.0542886Z (EngineCore_DP0 pid=3583) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:36.4546614Z FAILED 2025-10-10T01:42:36.4672775Z models/test_initialization.py::test_can_initialize_large_subset[Gemma3TextModel] Fork a new process to run a test 3587 2025-10-10T01:42:36.4682850Z Fork a new process to run a test 0 2025-10-10T01:42:36.4958261Z INFO 10-10 01:42:36 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma3TextModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/embeddinggemma-300m'} 2025-10-10T01:42:36.6111622Z 2025-10-10T01:42:36.6112674Z config.json: 0% 0.00/1.49k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:42:46.3759186Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] EngineCore failed to start. 2025-10-10T01:42:46.3759898Z 2025-10-10T01:42:46.3760825Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] Traceback (most recent call last): 2025-10-10T01:42:46.3761342Z 2025-10-10T01:42:46.3762308Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:46.3763002Z 2025-10-10T01:42:46.3763350Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:46.3763654Z 2025-10-10T01:42:46.3763941Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3764226Z 2025-10-10T01:42:46.3764722Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:46.3765176Z 2025-10-10T01:42:46.3765516Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:46.3765831Z 2025-10-10T01:42:46.3766303Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:46.3766891Z 2025-10-10T01:42:46.3767421Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:46.3767976Z 2025-10-10T01:42:46.3768411Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3768693Z 2025-10-10T01:42:46.3769201Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:46.3769676Z 2025-10-10T01:42:46.3769928Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] self._init_executor() 2025-10-10T01:42:46.3770190Z 2025-10-10T01:42:46.3770881Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:46.3771774Z 2025-10-10T01:42:46.3772617Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:46.3773160Z 2025-10-10T01:42:46.3773964Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:46.3774513Z 2025-10-10T01:42:46.3774884Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:46.3775233Z 2025-10-10T01:42:46.3775523Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3775812Z 2025-10-10T01:42:46.3776318Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:46.3776794Z 2025-10-10T01:42:46.3777062Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:42:46.3777344Z 2025-10-10T01:42:46.3777593Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3777853Z 2025-10-10T01:42:46.3778372Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:46.3778933Z 2025-10-10T01:42:46.3779238Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:46.3779534Z 2025-10-10T01:42:46.3779808Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3780088Z 2025-10-10T01:42:46.3780632Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:46.3781119Z 2025-10-10T01:42:46.3781422Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:42:46.3781729Z 2025-10-10T01:42:46.3781997Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3782279Z 2025-10-10T01:42:46.3782719Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:46.3783133Z 2025-10-10T01:42:46.3783469Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:46.3783847Z 2025-10-10T01:42:46.3784147Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3784424Z 2025-10-10T01:42:46.3784771Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:42:46.3785104Z 2025-10-10T01:42:46.3785458Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:42:46.3785795Z 2025-10-10T01:42:46.3786181Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:46.3786544Z 2025-10-10T01:42:46.3786891Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:42:46.3787234Z 2025-10-10T01:42:46.3787722Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:42:46.3788083Z 2025-10-10T01:42:46.3788456Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:46.3788817Z 2025-10-10T01:42:46.3789310Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:46.3789774Z 2025-10-10T01:42:46.3790136Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:46.3790467Z 2025-10-10T01:42:46.3790993Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:46.3791469Z 2025-10-10T01:42:46.3791843Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:46.3792198Z 2025-10-10T01:42:46.3792755Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:46.3793272Z 2025-10-10T01:42:46.3793569Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:42:46.3793921Z 2025-10-10T01:42:46.3794559Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:46.3795140Z 2025-10-10T01:42:46.3795508Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:46.3795871Z 2025-10-10T01:42:46.3796434Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3796721Z 2025-10-10T01:42:46.3797300Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:46.3797830Z 2025-10-10T01:42:46.3798131Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:42:46.3798426Z 2025-10-10T01:42:46.3798701Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3799165Z 2025-10-10T01:42:46.3799821Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:46.3800470Z 2025-10-10T01:42:46.3800807Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:42:46.3801147Z 2025-10-10T01:42:46.3801453Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3801774Z 2025-10-10T01:42:46.3802395Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:46.3802928Z 2025-10-10T01:42:46.3803243Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:46.3803564Z 2025-10-10T01:42:46.3803970Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3804254Z 2025-10-10T01:42:46.3804791Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:46.3805281Z 2025-10-10T01:42:46.3805568Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:42:46.3805852Z 2025-10-10T01:42:46.3806122Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3806391Z 2025-10-10T01:42:46.3806915Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:46.3807405Z 2025-10-10T01:42:46.3807728Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:42:46.3808040Z 2025-10-10T01:42:46.3808264Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:42:46.3808541Z 2025-10-10T01:42:46.3809023Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:46.3809557Z 2025-10-10T01:42:46.3809808Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] raise RuntimeError( 2025-10-10T01:42:46.3810072Z 2025-10-10T01:42:46.3810636Z (EngineCore_DP0 pid=3666) ERROR 10-10 01:42:46 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:46.3811343Z (EngineCore_DP0 pid=3666) Process EngineCore_DP0: 2025-10-10T01:42:46.3811745Z (EngineCore_DP0 pid=3666) Traceback (most recent call last): 2025-10-10T01:42:46.3812367Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:42:46.3812884Z (EngineCore_DP0 pid=3666) self.run() 2025-10-10T01:42:46.3813414Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:42:46.3813986Z (EngineCore_DP0 pid=3666) self._target(*self._args, **self._kwargs) 2025-10-10T01:42:46.3814658Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:42:46.3815209Z (EngineCore_DP0 pid=3666) raise e 2025-10-10T01:42:46.3815849Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:42:46.3816483Z (EngineCore_DP0 pid=3666) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:42:46.3816923Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3817547Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:42:46.3818195Z (EngineCore_DP0 pid=3666) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:42:46.3818863Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:42:46.3819484Z (EngineCore_DP0 pid=3666) self.model_executor = executor_class(vllm_config) 2025-10-10T01:42:46.3820109Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3820893Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:42:46.3821546Z (EngineCore_DP0 pid=3666) self._init_executor() 2025-10-10T01:42:46.3822213Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:42:46.3822898Z (EngineCore_DP0 pid=3666) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:42:46.3823623Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:42:46.3824336Z (EngineCore_DP0 pid=3666) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:42:46.3824833Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3825476Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:42:46.3826060Z (EngineCore_DP0 pid=3666) return func(*args, **kwargs) 2025-10-10T01:42:46.3826443Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3827079Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:42:46.3827756Z (EngineCore_DP0 pid=3666) worker_class = resolve_obj_by_qualname( 2025-10-10T01:42:46.3828177Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3828841Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:42:46.3829518Z (EngineCore_DP0 pid=3666) module = importlib.import_module(module_name) 2025-10-10T01:42:46.3829965Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3830541Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:42:46.3831165Z (EngineCore_DP0 pid=3666) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:42:46.3831657Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3832153Z (EngineCore_DP0 pid=3666) File "", line 1387, in _gcd_import 2025-10-10T01:42:46.3832712Z (EngineCore_DP0 pid=3666) File "", line 1360, in _find_and_load 2025-10-10T01:42:46.3833367Z (EngineCore_DP0 pid=3666) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:42:46.3833936Z (EngineCore_DP0 pid=3666) File "", line 935, in _load_unlocked 2025-10-10T01:42:46.3834504Z (EngineCore_DP0 pid=3666) File "", line 999, in exec_module 2025-10-10T01:42:46.3835103Z (EngineCore_DP0 pid=3666) File "", line 488, in _call_with_frames_removed 2025-10-10T01:42:46.3835846Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:42:46.3836532Z (EngineCore_DP0 pid=3666) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:42:46.3837717Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:42:46.3838569Z (EngineCore_DP0 pid=3666) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:42:46.3839493Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:42:46.3840175Z (EngineCore_DP0 pid=3666) class FlashAttentionMetadataBuilder( 2025-10-10T01:42:46.3840980Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:42:46.3841808Z (EngineCore_DP0 pid=3666) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:42:46.3842294Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3843019Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:42:46.3843709Z (EngineCore_DP0 pid=3666) if not is_fa_version_supported(fa_version): 2025-10-10T01:42:46.3844152Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3844896Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:42:46.3845591Z (EngineCore_DP0 pid=3666) return _is_fa2_supported(device)[0] 2025-10-10T01:42:46.3846059Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3846771Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:42:46.3847494Z (EngineCore_DP0 pid=3666) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:42:46.3847951Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3848616Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:42:46.3849254Z (EngineCore_DP0 pid=3666) prop = get_device_properties(device) 2025-10-10T01:42:46.3849669Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:42:46.3850344Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:42:46.3851004Z (EngineCore_DP0 pid=3666) _lazy_init() # will define _get_device_properties 2025-10-10T01:42:46.3851388Z (EngineCore_DP0 pid=3666) ^^^^^^^^^^^^ 2025-10-10T01:42:46.3852048Z (EngineCore_DP0 pid=3666) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:42:46.3852605Z (EngineCore_DP0 pid=3666) raise RuntimeError( 2025-10-10T01:42:46.3853288Z (EngineCore_DP0 pid=3666) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:42:46.8141594Z FAILED 2025-10-10T01:42:46.8268643Z models/test_initialization.py::test_can_initialize_large_subset[Plamo2ForCausalLM] Fork a new process to run a test 3670 2025-10-10T01:42:46.8279705Z Fork a new process to run a test 0 2025-10-10T01:42:46.8283716Z `transformers==4.56.2` installed, but `transformers<=4.55.4` is required to run this model. Reason: HF model uses remote code that is not compatible with latest Transformers 2025-10-10T01:42:47.1256984Z PASSED 2025-10-10T01:42:47.1383331Z models/test_initialization.py::test_can_initialize_large_subset[WhisperForConditionalGeneration] Fork a new process to run a test 3671 2025-10-10T01:42:47.1392863Z Fork a new process to run a test 0 2025-10-10T01:42:47.1669406Z INFO 10-10 01:42:47 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='WhisperForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'openai/whisper-large-v3'} 2025-10-10T01:42:47.2574443Z 2025-10-10T01:42:47.2576825Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:42:47.2577160Z config.json: 1.27kB [00:00, 6.83MB/s] 2025-10-10T01:42:47.3944383Z 2025-10-10T01:42:47.3945987Z preprocessor_config.json: 0% 0.00/340 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:43:03.9752674Z [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 2025-10-10T01:43:03.9780840Z [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 2025-10-10T01:43:03.9788602Z [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 2025-10-10T01:43:03.9796934Z [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 2025-10-10T01:43:03.9804543Z [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 2025-10-10T01:43:03.9811745Z [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 2025-10-10T01:43:03.9814566Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:03 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 2025-10-10T01:43:04.2225015Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:04 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. 2025-10-10T01:43:05.8505214Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:05 [gpu_model_runner.py:2707] Starting to load model openai/whisper-large-v3... 2025-10-10T01:43:06.1132727Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:06 [gpu_model_runner.py:2739] Loading model from scratch... 2025-10-10T01:43:06.1160815Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:06 [layer.py:477] MultiHeadAttention attn_backend: _Backend.FLASH_ATTN, use_upstream_fa: False 2025-10-10T01:43:06.1732085Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:06 [cuda.py:361] Using Flash Attention backend on V1 engine. 2025-10-10T01:43:06.2511643Z (EngineCore_DP0 pid=3753) WARNING 10-10 01:43:06 [vllm.py:754] `torch.compile` is turned on, but the model openai/whisper-large-v3 does not support it. Please open an issue on GitHub if you want it to be supported. 2025-10-10T01:43:06.7395435Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:06 [gpu_model_runner.py:2758] Model loading took 2.8763 GiB and 0.158210 seconds 2025-10-10T01:43:06.9671050Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:06 [gpu_model_runner.py:3447] Encoder cache will be initialized with a budget of 1500 tokens, and profiled with 1 audio items of the maximum feature size. 2025-10-10T01:43:06.9688580Z (EngineCore_DP0 pid=3753) WARNING 10-10 01:43:06 [processing.py:1089] WhisperProcessor did not return `BatchFeature`. Make sure to match the behaviour of `ProcessorMixin` when implementing custom processors. 2025-10-10T01:43:08.0452958Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:08 [gpu_worker.py:298] Available KV cache memory: 14.45 GiB 2025-10-10T01:43:08.2726283Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:08 [kv_cache_utils.py:1087] GPU KV cache size: 47,328 tokens 2025-10-10T01:43:08.2727193Z (EngineCore_DP0 pid=3753) INFO 10-10 01:43:08 [kv_cache_utils.py:1091] Maximum concurrency for 448 tokens per request: 48.50x 2025-10-10T01:43:08.5175732Z (EngineCore_DP0 pid=3753) 2025-10-10T01:43:08.6502107Z Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0% 0/67 [00:00, model_arch='MiniCPMForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'openbmb/MiniCPM-2B-sft-bf16'} 2025-10-10T01:43:12.2767200Z 2025-10-10T01:43:12.2769304Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:43:12.2769701Z config.json: 1.01kB [00:00, 5.49MB/s] 2025-10-10T01:43:12.3506915Z 2025-10-10T01:43:12.3508813Z configuration_minicpm.py: 0.00B [00:00, ?B/s] 2025-10-10T01:43:12.3509293Z configuration_minicpm.py: 9.77kB [00:00, 64.9MB/s] 2025-10-10T01:43:12.3600597Z A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-2B-sft-bf16: 2025-10-10T01:43:12.3601234Z - configuration_minicpm.py 2025-10-10T01:43:12.3601861Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:43:19.1225707Z INFO 10-10 01:43:19 [model.py:551] Resolved architecture: MiniCPMForCausalLM 2025-10-10T01:43:19.1226196Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:43:19.1471217Z INFO 10-10 01:43:19 [model.py:1545] Using max model len 4096 2025-10-10T01:43:19.1472819Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:43:19.2070567Z INFO 10-10 01:43:19 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:43:19.2663712Z 2025-10-10T01:43:19.2665391Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:43:19.2665948Z tokenizer_config.json: 1.12kB [00:00, 9.31MB/s] 2025-10-10T01:43:19.3923817Z 2025-10-10T01:43:19.6410916Z tokenizer.model: 0% 0.00/1.99M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:43:20.3310314Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] EngineCore failed to start. 2025-10-10T01:43:20.3310739Z 2025-10-10T01:43:20.3311168Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] Traceback (most recent call last): 2025-10-10T01:43:20.3311542Z 2025-10-10T01:43:20.3312243Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:20.3312836Z 2025-10-10T01:43:20.3313232Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:20.3313612Z 2025-10-10T01:43:20.3313966Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3314546Z 2025-10-10T01:43:20.3315171Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:20.3315733Z 2025-10-10T01:43:20.3316166Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:20.3316567Z 2025-10-10T01:43:20.3317150Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:20.3317702Z 2025-10-10T01:43:20.3318097Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:20.3318476Z 2025-10-10T01:43:20.3318839Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3319305Z 2025-10-10T01:43:20.3319957Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:20.3320656Z 2025-10-10T01:43:20.3321034Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] self._init_executor() 2025-10-10T01:43:20.3321343Z 2025-10-10T01:43:20.3322058Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:20.3322581Z 2025-10-10T01:43:20.3323115Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:20.3323528Z 2025-10-10T01:43:20.3324211Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:20.3324726Z 2025-10-10T01:43:20.3325320Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:20.3325674Z 2025-10-10T01:43:20.3326249Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3326605Z 2025-10-10T01:43:20.3327143Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:20.3327610Z 2025-10-10T01:43:20.3327885Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:43:20.3328163Z 2025-10-10T01:43:20.3328419Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3328675Z 2025-10-10T01:43:20.3329202Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:20.3329683Z 2025-10-10T01:43:20.3329983Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:20.3330267Z 2025-10-10T01:43:20.3330534Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3330800Z 2025-10-10T01:43:20.3331356Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:20.3331933Z 2025-10-10T01:43:20.3332255Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:43:20.3332574Z 2025-10-10T01:43:20.3332864Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3333149Z 2025-10-10T01:43:20.3333600Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:20.3334017Z 2025-10-10T01:43:20.3334368Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:20.3334700Z 2025-10-10T01:43:20.3335000Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3335281Z 2025-10-10T01:43:20.3335633Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:43:20.3335967Z 2025-10-10T01:43:20.3336331Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:43:20.3336713Z 2025-10-10T01:43:20.3337106Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:20.3337470Z 2025-10-10T01:43:20.3337813Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:43:20.3338147Z 2025-10-10T01:43:20.3338504Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:43:20.3338860Z 2025-10-10T01:43:20.3339236Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:20.3339594Z 2025-10-10T01:43:20.3340096Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:20.3340558Z 2025-10-10T01:43:20.3341004Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:20.3341361Z 2025-10-10T01:43:20.3341919Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:20.3342398Z 2025-10-10T01:43:20.3342789Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:20.3343158Z 2025-10-10T01:43:20.3343730Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:20.3344245Z 2025-10-10T01:43:20.3344551Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:43:20.3344854Z 2025-10-10T01:43:20.3345488Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:20.3346059Z 2025-10-10T01:43:20.3346457Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:20.3346810Z 2025-10-10T01:43:20.3347140Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3347414Z 2025-10-10T01:43:20.3348005Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:20.3348533Z 2025-10-10T01:43:20.3348860Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:43:20.3349169Z 2025-10-10T01:43:20.3349455Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3349733Z 2025-10-10T01:43:20.3350352Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:20.3350908Z 2025-10-10T01:43:20.3351207Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:43:20.3351491Z 2025-10-10T01:43:20.3351765Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3352078Z 2025-10-10T01:43:20.3352682Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:20.3353211Z 2025-10-10T01:43:20.3353537Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:20.3353855Z 2025-10-10T01:43:20.3354135Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3354417Z 2025-10-10T01:43:20.3354951Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:20.3355439Z 2025-10-10T01:43:20.3355728Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:43:20.3356019Z 2025-10-10T01:43:20.3356398Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3356684Z 2025-10-10T01:43:20.3357215Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:20.3357712Z 2025-10-10T01:43:20.3358041Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:43:20.3358346Z 2025-10-10T01:43:20.3358584Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:43:20.3358825Z 2025-10-10T01:43:20.3359450Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:20.3359909Z 2025-10-10T01:43:20.3360175Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] raise RuntimeError( 2025-10-10T01:43:20.3360440Z 2025-10-10T01:43:20.3361016Z (EngineCore_DP0 pid=3966) ERROR 10-10 01:43:20 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:20.3361730Z (EngineCore_DP0 pid=3966) Process EngineCore_DP0: 2025-10-10T01:43:20.3362143Z (EngineCore_DP0 pid=3966) Traceback (most recent call last): 2025-10-10T01:43:20.3362828Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:43:20.3363359Z (EngineCore_DP0 pid=3966) self.run() 2025-10-10T01:43:20.3363890Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:43:20.3364481Z (EngineCore_DP0 pid=3966) self._target(*self._args, **self._kwargs) 2025-10-10T01:43:20.3365151Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:43:20.3365699Z (EngineCore_DP0 pid=3966) raise e 2025-10-10T01:43:20.3366311Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:20.3366954Z (EngineCore_DP0 pid=3966) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:20.3367413Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3368045Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:20.3368751Z (EngineCore_DP0 pid=3966) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:20.3369427Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:20.3370059Z (EngineCore_DP0 pid=3966) self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:20.3370520Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3371185Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:20.3371786Z (EngineCore_DP0 pid=3966) self._init_executor() 2025-10-10T01:43:20.3372465Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:20.3373155Z (EngineCore_DP0 pid=3966) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:20.3373963Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:20.3374696Z (EngineCore_DP0 pid=3966) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:20.3375203Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3375847Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:20.3376438Z (EngineCore_DP0 pid=3966) return func(*args, **kwargs) 2025-10-10T01:43:20.3376819Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3377455Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:20.3378093Z (EngineCore_DP0 pid=3966) worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:20.3378521Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3379189Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:20.3379851Z (EngineCore_DP0 pid=3966) module = importlib.import_module(module_name) 2025-10-10T01:43:20.3380310Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3380952Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:20.3381604Z (EngineCore_DP0 pid=3966) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:20.3382108Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3382599Z (EngineCore_DP0 pid=3966) File "", line 1387, in _gcd_import 2025-10-10T01:43:20.3383167Z (EngineCore_DP0 pid=3966) File "", line 1360, in _find_and_load 2025-10-10T01:43:20.3383762Z (EngineCore_DP0 pid=3966) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:20.3384356Z (EngineCore_DP0 pid=3966) File "", line 935, in _load_unlocked 2025-10-10T01:43:20.3384932Z (EngineCore_DP0 pid=3966) File "", line 999, in exec_module 2025-10-10T01:43:20.3385536Z (EngineCore_DP0 pid=3966) File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:20.3386339Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:20.3387035Z (EngineCore_DP0 pid=3966) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:20.3387759Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:20.3388505Z (EngineCore_DP0 pid=3966) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:20.3389288Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:20.3389972Z (EngineCore_DP0 pid=3966) class FlashAttentionMetadataBuilder( 2025-10-10T01:43:20.3390774Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:20.3391676Z (EngineCore_DP0 pid=3966) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:20.3392188Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3392894Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:20.3393588Z (EngineCore_DP0 pid=3966) if not is_fa_version_supported(fa_version): 2025-10-10T01:43:20.3394024Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3394764Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:20.3395474Z (EngineCore_DP0 pid=3966) return _is_fa2_supported(device)[0] 2025-10-10T01:43:20.3395896Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3396830Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:20.3397558Z (EngineCore_DP0 pid=3966) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:20.3398014Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3398692Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:20.3399505Z (EngineCore_DP0 pid=3966) prop = get_device_properties(device) 2025-10-10T01:43:20.3399943Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:20.3400609Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:20.3401279Z (EngineCore_DP0 pid=3966) _lazy_init() # will define _get_device_properties 2025-10-10T01:43:20.3401698Z (EngineCore_DP0 pid=3966) ^^^^^^^^^^^^ 2025-10-10T01:43:20.3402297Z (EngineCore_DP0 pid=3966) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:20.3402875Z (EngineCore_DP0 pid=3966) raise RuntimeError( 2025-10-10T01:43:20.3403567Z (EngineCore_DP0 pid=3966) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:20.7358278Z FAILED 2025-10-10T01:43:20.7486202Z models/test_initialization.py::test_can_initialize_large_subset[NemotronHForCausalLM] Fork a new process to run a test 3970 2025-10-10T01:43:20.7497398Z Fork a new process to run a test 0 2025-10-10T01:43:20.7773863Z INFO 10-10 01:43:20 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='NemotronHForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nvidia/Nemotron-H-8B-Base-8K'} 2025-10-10T01:43:21.5277038Z 2025-10-10T01:43:21.5279005Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:43:21.5279579Z config.json: 1.50kB [00:00, 7.62MB/s] 2025-10-10T01:43:21.7084331Z 2025-10-10T01:43:21.7086493Z configuration_nemotron_h.py: 0.00B [00:00, ?B/s] 2025-10-10T01:43:21.7087056Z configuration_nemotron_h.py: 12.1kB [00:00, 45.3MB/s] 2025-10-10T01:43:21.7179875Z A new version of the following files was downloaded from https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K: 2025-10-10T01:43:21.7180559Z - configuration_nemotron_h.py 2025-10-10T01:43:21.7181469Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:43:28.5970487Z INFO 10-10 01:43:28 [model.py:551] Resolved architecture: NemotronHForCausalLM 2025-10-10T01:43:28.5971001Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:43:28.6223096Z INFO 10-10 01:43:28 [model.py:1545] Using max model len 8192 2025-10-10T01:43:28.6225025Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:43:28.6833801Z INFO 10-10 01:43:28 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:43:28.6836116Z INFO 10-10 01:43:28 [config.py:297] Hybrid or mamba-based model detected: disabling prefix caching since it is not yet supported. 2025-10-10T01:43:28.6836994Z INFO 10-10 01:43:28 [config.py:308] Hybrid or mamba-based model detected: setting cudagraph mode to FULL_AND_PIECEWISE in order to optimize performance. 2025-10-10T01:43:28.7283683Z INFO 10-10 01:43:28 [config.py:376] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. 2025-10-10T01:43:28.7286129Z INFO 10-10 01:43:28 [config.py:397] Padding mamba page size by 0.19% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:43:28.7664023Z 2025-10-10T01:43:28.7669260Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:43:28.7669674Z tokenizer_config.json: 178kB [00:00, 367MB/s] 2025-10-10T01:43:28.8750335Z 2025-10-10T01:43:29.1063453Z tokenizer.json: 0% 0.00/17.1M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:43:29.9578874Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] EngineCore failed to start. 2025-10-10T01:43:29.9579368Z 2025-10-10T01:43:29.9579779Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] Traceback (most recent call last): 2025-10-10T01:43:29.9580149Z 2025-10-10T01:43:29.9580682Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:29.9581195Z 2025-10-10T01:43:29.9581509Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:29.9581814Z 2025-10-10T01:43:29.9582089Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9582366Z 2025-10-10T01:43:29.9582849Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:29.9583588Z 2025-10-10T01:43:29.9583927Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:29.9584242Z 2025-10-10T01:43:29.9584784Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:29.9585311Z 2025-10-10T01:43:29.9585679Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:29.9585987Z 2025-10-10T01:43:29.9586276Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9586562Z 2025-10-10T01:43:29.9587068Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:29.9587533Z 2025-10-10T01:43:29.9587788Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] self._init_executor() 2025-10-10T01:43:29.9588044Z 2025-10-10T01:43:29.9588604Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:29.9589685Z 2025-10-10T01:43:29.9590363Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:29.9590947Z 2025-10-10T01:43:29.9591747Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:29.9592541Z 2025-10-10T01:43:29.9593030Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:29.9593464Z 2025-10-10T01:43:29.9593776Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9594058Z 2025-10-10T01:43:29.9594764Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:29.9595257Z 2025-10-10T01:43:29.9595546Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:43:29.9595817Z 2025-10-10T01:43:29.9596317Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9596640Z 2025-10-10T01:43:29.9597174Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:29.9597660Z 2025-10-10T01:43:29.9597944Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:29.9598235Z 2025-10-10T01:43:29.9598498Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9598768Z 2025-10-10T01:43:29.9599398Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:29.9599890Z 2025-10-10T01:43:29.9600191Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:43:29.9600489Z 2025-10-10T01:43:29.9600766Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9601149Z 2025-10-10T01:43:29.9601603Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:29.9602015Z 2025-10-10T01:43:29.9602374Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:29.9602700Z 2025-10-10T01:43:29.9602985Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9603280Z 2025-10-10T01:43:29.9603619Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:43:29.9603953Z 2025-10-10T01:43:29.9604292Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:43:29.9604630Z 2025-10-10T01:43:29.9605016Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:29.9605478Z 2025-10-10T01:43:29.9605838Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:43:29.9606169Z 2025-10-10T01:43:29.9606537Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:43:29.9606882Z 2025-10-10T01:43:29.9607270Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:29.9607624Z 2025-10-10T01:43:29.9608124Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:29.9608585Z 2025-10-10T01:43:29.9608934Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:29.9609275Z 2025-10-10T01:43:29.9610023Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:29.9610620Z 2025-10-10T01:43:29.9611075Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:29.9611510Z 2025-10-10T01:43:29.9612104Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:29.9612615Z 2025-10-10T01:43:29.9612902Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:43:29.9613196Z 2025-10-10T01:43:29.9613838Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:29.9614410Z 2025-10-10T01:43:29.9614785Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:29.9615144Z 2025-10-10T01:43:29.9615404Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9615683Z 2025-10-10T01:43:29.9616261Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:29.9616831Z 2025-10-10T01:43:29.9617126Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:43:29.9617435Z 2025-10-10T01:43:29.9617709Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9617990Z 2025-10-10T01:43:29.9618582Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:29.9619130Z 2025-10-10T01:43:29.9619428Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:43:29.9619764Z 2025-10-10T01:43:29.9620074Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9620385Z 2025-10-10T01:43:29.9621069Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:29.9621673Z 2025-10-10T01:43:29.9622014Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:29.9622322Z 2025-10-10T01:43:29.9622597Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9622866Z 2025-10-10T01:43:29.9623386Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:29.9623872Z 2025-10-10T01:43:29.9624157Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:43:29.9624452Z 2025-10-10T01:43:29.9624721Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9625005Z 2025-10-10T01:43:29.9625609Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:29.9626103Z 2025-10-10T01:43:29.9626426Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:43:29.9626727Z 2025-10-10T01:43:29.9626965Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:43:29.9627208Z 2025-10-10T01:43:29.9627695Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:29.9628140Z 2025-10-10T01:43:29.9628387Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] raise RuntimeError( 2025-10-10T01:43:29.9628640Z 2025-10-10T01:43:29.9629201Z (EngineCore_DP0 pid=4048) ERROR 10-10 01:43:29 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:29.9629896Z (EngineCore_DP0 pid=4048) Process EngineCore_DP0: 2025-10-10T01:43:29.9630307Z (EngineCore_DP0 pid=4048) Traceback (most recent call last): 2025-10-10T01:43:29.9630907Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:43:29.9631429Z (EngineCore_DP0 pid=4048) self.run() 2025-10-10T01:43:29.9631965Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:43:29.9632582Z (EngineCore_DP0 pid=4048) self._target(*self._args, **self._kwargs) 2025-10-10T01:43:29.9633245Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:43:29.9633806Z (EngineCore_DP0 pid=4048) raise e 2025-10-10T01:43:29.9634409Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:29.9635042Z (EngineCore_DP0 pid=4048) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:29.9635479Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9636088Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:29.9636721Z (EngineCore_DP0 pid=4048) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:29.9637420Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:29.9638106Z (EngineCore_DP0 pid=4048) self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:29.9638562Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9639264Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:29.9639859Z (EngineCore_DP0 pid=4048) self._init_executor() 2025-10-10T01:43:29.9640521Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:29.9641210Z (EngineCore_DP0 pid=4048) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:29.9641941Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:29.9642731Z (EngineCore_DP0 pid=4048) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:29.9643242Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9643897Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:29.9644495Z (EngineCore_DP0 pid=4048) return func(*args, **kwargs) 2025-10-10T01:43:29.9644874Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9645505Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:29.9646148Z (EngineCore_DP0 pid=4048) worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:29.9646591Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9647277Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:29.9647938Z (EngineCore_DP0 pid=4048) module = importlib.import_module(module_name) 2025-10-10T01:43:29.9648375Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9648960Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:29.9649582Z (EngineCore_DP0 pid=4048) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:29.9650142Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9650644Z (EngineCore_DP0 pid=4048) File "", line 1387, in _gcd_import 2025-10-10T01:43:29.9651204Z (EngineCore_DP0 pid=4048) File "", line 1360, in _find_and_load 2025-10-10T01:43:29.9651791Z (EngineCore_DP0 pid=4048) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:29.9652380Z (EngineCore_DP0 pid=4048) File "", line 935, in _load_unlocked 2025-10-10T01:43:29.9652949Z (EngineCore_DP0 pid=4048) File "", line 999, in exec_module 2025-10-10T01:43:29.9653548Z (EngineCore_DP0 pid=4048) File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:29.9654276Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:29.9654940Z (EngineCore_DP0 pid=4048) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:29.9655721Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:29.9656432Z (EngineCore_DP0 pid=4048) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:29.9657207Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:29.9657882Z (EngineCore_DP0 pid=4048) class FlashAttentionMetadataBuilder( 2025-10-10T01:43:29.9658658Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:29.9659466Z (EngineCore_DP0 pid=4048) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:29.9659962Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9660736Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:29.9661442Z (EngineCore_DP0 pid=4048) if not is_fa_version_supported(fa_version): 2025-10-10T01:43:29.9661891Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9662639Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:29.9663330Z (EngineCore_DP0 pid=4048) return _is_fa2_supported(device)[0] 2025-10-10T01:43:29.9663742Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9664450Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:29.9665166Z (EngineCore_DP0 pid=4048) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:29.9665614Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9666283Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:29.9666928Z (EngineCore_DP0 pid=4048) prop = get_device_properties(device) 2025-10-10T01:43:29.9667347Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:29.9668069Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:29.9668730Z (EngineCore_DP0 pid=4048) _lazy_init() # will define _get_device_properties 2025-10-10T01:43:29.9669130Z (EngineCore_DP0 pid=4048) ^^^^^^^^^^^^ 2025-10-10T01:43:29.9669708Z (EngineCore_DP0 pid=4048) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:29.9670277Z (EngineCore_DP0 pid=4048) raise RuntimeError( 2025-10-10T01:43:29.9670959Z (EngineCore_DP0 pid=4048) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:30.3760825Z FAILED 2025-10-10T01:43:30.3888357Z models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5_MoeForCausalLM] Fork a new process to run a test 4052 2025-10-10T01:43:30.3899344Z Fork a new process to run a test 0 2025-10-10T01:43:30.4175372Z INFO 10-10 01:43:30 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Ernie4_5_MoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'baidu/ERNIE-4.5-21B-A3B-PT'} 2025-10-10T01:43:30.5693589Z INFO 10-10 01:43:30 [model.py:551] Resolved architecture: Ernie4_5_MoeForCausalLM 2025-10-10T01:43:30.5694053Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:43:30.5941290Z INFO 10-10 01:43:30 [model.py:1545] Using max model len 131072 2025-10-10T01:43:30.7640781Z INFO 10-10 01:43:30 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:43:31.9440539Z (EngineCore_DP0 pid=4059) INFO 10-10 01:43:31 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:43:31.9518498Z (EngineCore_DP0 pid=4059) INFO 10-10 01:43:31 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='baidu/ERNIE-4.5-21B-A3B-PT', speculative_config=None, tokenizer='baidu/ERNIE-4.5-21B-A3B-PT', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=baidu/ERNIE-4.5-21B-A3B-PT, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:43:32.0748977Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] EngineCore failed to start. 2025-10-10T01:43:32.0749698Z 2025-10-10T01:43:32.0750180Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] Traceback (most recent call last): 2025-10-10T01:43:32.0750569Z 2025-10-10T01:43:32.0751269Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:32.0751880Z 2025-10-10T01:43:32.0752209Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:32.0752514Z 2025-10-10T01:43:32.0752792Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0753066Z 2025-10-10T01:43:32.0753585Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:32.0754241Z 2025-10-10T01:43:32.0754607Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:32.0754929Z 2025-10-10T01:43:32.0755564Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:32.0756015Z 2025-10-10T01:43:32.0756351Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:32.0756670Z 2025-10-10T01:43:32.0756951Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0757229Z 2025-10-10T01:43:32.0757738Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:32.0758228Z 2025-10-10T01:43:32.0758655Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] self._init_executor() 2025-10-10T01:43:32.0759216Z 2025-10-10T01:43:32.0760042Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:32.0760986Z 2025-10-10T01:43:32.0761588Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:32.0762223Z 2025-10-10T01:43:32.0763094Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:32.0763772Z 2025-10-10T01:43:32.0764177Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:32.0764528Z 2025-10-10T01:43:32.0764822Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0765112Z 2025-10-10T01:43:32.0765611Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:32.0766071Z 2025-10-10T01:43:32.0766337Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:43:32.0766750Z 2025-10-10T01:43:32.0767010Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0767270Z 2025-10-10T01:43:32.0767793Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:32.0768282Z 2025-10-10T01:43:32.0768583Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:32.0768890Z 2025-10-10T01:43:32.0769167Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0769440Z 2025-10-10T01:43:32.0769979Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:32.0770475Z 2025-10-10T01:43:32.0770777Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:43:32.0771087Z 2025-10-10T01:43:32.0771356Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0771679Z 2025-10-10T01:43:32.0772127Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:32.0772544Z 2025-10-10T01:43:32.0772883Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:32.0773212Z 2025-10-10T01:43:32.0773504Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0773783Z 2025-10-10T01:43:32.0774127Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:43:32.0774457Z 2025-10-10T01:43:32.0774811Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:43:32.0775146Z 2025-10-10T01:43:32.0775596Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:32.0775977Z 2025-10-10T01:43:32.0776325Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:43:32.0776664Z 2025-10-10T01:43:32.0777025Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:43:32.0777386Z 2025-10-10T01:43:32.0777758Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:32.0778123Z 2025-10-10T01:43:32.0778623Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:32.0779087Z 2025-10-10T01:43:32.0779448Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:32.0779854Z 2025-10-10T01:43:32.0780392Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:32.0780866Z 2025-10-10T01:43:32.0781255Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:32.0781661Z 2025-10-10T01:43:32.0782227Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:32.0782737Z 2025-10-10T01:43:32.0783029Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:43:32.0783330Z 2025-10-10T01:43:32.0783953Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:32.0784528Z 2025-10-10T01:43:32.0784889Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:32.0785246Z 2025-10-10T01:43:32.0785498Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0785763Z 2025-10-10T01:43:32.0786322Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:32.0786879Z 2025-10-10T01:43:32.0787196Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:43:32.0787493Z 2025-10-10T01:43:32.0787770Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0788044Z 2025-10-10T01:43:32.0788661Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:32.0789218Z 2025-10-10T01:43:32.0789497Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:43:32.0789784Z 2025-10-10T01:43:32.0790040Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0790313Z 2025-10-10T01:43:32.0790962Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:32.0791499Z 2025-10-10T01:43:32.0791819Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:32.0792143Z 2025-10-10T01:43:32.0792415Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0792688Z 2025-10-10T01:43:32.0793217Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:32.0793698Z 2025-10-10T01:43:32.0793980Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:43:32.0794265Z 2025-10-10T01:43:32.0794536Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0794809Z 2025-10-10T01:43:32.0795332Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:32.0795819Z 2025-10-10T01:43:32.0796373Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:43:32.0796791Z 2025-10-10T01:43:32.0797034Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:43:32.0797286Z 2025-10-10T01:43:32.0797773Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:32.0798237Z 2025-10-10T01:43:32.0798485Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] raise RuntimeError( 2025-10-10T01:43:32.0798756Z 2025-10-10T01:43:32.0799442Z (EngineCore_DP0 pid=4059) ERROR 10-10 01:43:32 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:32.0800144Z (EngineCore_DP0 pid=4059) Process EngineCore_DP0: 2025-10-10T01:43:32.0800567Z (EngineCore_DP0 pid=4059) Traceback (most recent call last): 2025-10-10T01:43:32.0801174Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:43:32.0801690Z (EngineCore_DP0 pid=4059) self.run() 2025-10-10T01:43:32.0802216Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:43:32.0802861Z (EngineCore_DP0 pid=4059) self._target(*self._args, **self._kwargs) 2025-10-10T01:43:32.0803529Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:43:32.0804079Z (EngineCore_DP0 pid=4059) raise e 2025-10-10T01:43:32.0804664Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:32.0811166Z (EngineCore_DP0 pid=4059) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:32.0811766Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0812456Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:32.0813327Z (EngineCore_DP0 pid=4059) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:32.0814043Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:32.0814682Z (EngineCore_DP0 pid=4059) self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:32.0815147Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0815807Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:32.0816391Z (EngineCore_DP0 pid=4059) self._init_executor() 2025-10-10T01:43:32.0817064Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:32.0817762Z (EngineCore_DP0 pid=4059) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:32.0818505Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:32.0819221Z (EngineCore_DP0 pid=4059) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:32.0819714Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0820360Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:32.0821003Z (EngineCore_DP0 pid=4059) return func(*args, **kwargs) 2025-10-10T01:43:32.0821393Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0822033Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:32.0822664Z (EngineCore_DP0 pid=4059) worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:32.0823086Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0823756Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:32.0824416Z (EngineCore_DP0 pid=4059) module = importlib.import_module(module_name) 2025-10-10T01:43:32.0824861Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0825443Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:32.0826118Z (EngineCore_DP0 pid=4059) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:32.0826620Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0827131Z (EngineCore_DP0 pid=4059) File "", line 1387, in _gcd_import 2025-10-10T01:43:32.0827689Z (EngineCore_DP0 pid=4059) File "", line 1360, in _find_and_load 2025-10-10T01:43:32.0828281Z (EngineCore_DP0 pid=4059) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:32.0828851Z (EngineCore_DP0 pid=4059) File "", line 935, in _load_unlocked 2025-10-10T01:43:32.0829429Z (EngineCore_DP0 pid=4059) File "", line 999, in exec_module 2025-10-10T01:43:32.0830043Z (EngineCore_DP0 pid=4059) File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:32.0830850Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:32.0831544Z (EngineCore_DP0 pid=4059) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:32.0832273Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:32.0833008Z (EngineCore_DP0 pid=4059) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:32.0833798Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:32.0834472Z (EngineCore_DP0 pid=4059) class FlashAttentionMetadataBuilder( 2025-10-10T01:43:32.0835272Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:32.0836088Z (EngineCore_DP0 pid=4059) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:32.0836566Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0837259Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:32.0837943Z (EngineCore_DP0 pid=4059) if not is_fa_version_supported(fa_version): 2025-10-10T01:43:32.0838434Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0839272Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:32.0839980Z (EngineCore_DP0 pid=4059) return _is_fa2_supported(device)[0] 2025-10-10T01:43:32.0840393Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0841106Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:32.0841835Z (EngineCore_DP0 pid=4059) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:32.0842294Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0842966Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:32.0843605Z (EngineCore_DP0 pid=4059) prop = get_device_properties(device) 2025-10-10T01:43:32.0844081Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:32.0844769Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:32.0845433Z (EngineCore_DP0 pid=4059) _lazy_init() # will define _get_device_properties 2025-10-10T01:43:32.0845825Z (EngineCore_DP0 pid=4059) ^^^^^^^^^^^^ 2025-10-10T01:43:32.0846451Z (EngineCore_DP0 pid=4059) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:32.0847041Z (EngineCore_DP0 pid=4059) raise RuntimeError( 2025-10-10T01:43:32.0847738Z (EngineCore_DP0 pid=4059) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:32.4812812Z FAILED 2025-10-10T01:43:32.4940018Z models/test_initialization.py::test_can_initialize_large_subset[MambaForCausalLM] Fork a new process to run a test 4063 2025-10-10T01:43:32.4951521Z Fork a new process to run a test 0 2025-10-10T01:43:32.5222418Z INFO 10-10 01:43:32 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MambaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'state-spaces/mamba-130m-hf'} 2025-10-10T01:43:32.6295729Z 2025-10-10T01:43:32.6297162Z config.json: 0% 0.00/895 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:43:40.0899148Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] EngineCore failed to start. 2025-10-10T01:43:40.0899595Z 2025-10-10T01:43:40.0900004Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] Traceback (most recent call last): 2025-10-10T01:43:40.0900361Z 2025-10-10T01:43:40.0901043Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:40.0901634Z 2025-10-10T01:43:40.0902030Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:40.0902409Z 2025-10-10T01:43:40.0902774Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0903325Z 2025-10-10T01:43:40.0903948Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:40.0904496Z 2025-10-10T01:43:40.0904927Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:40.0905346Z 2025-10-10T01:43:40.0905945Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:40.0906495Z 2025-10-10T01:43:40.0906883Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:40.0907270Z 2025-10-10T01:43:40.0907614Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0907964Z 2025-10-10T01:43:40.0908627Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:40.0909306Z 2025-10-10T01:43:40.0909572Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] self._init_executor() 2025-10-10T01:43:40.0909842Z 2025-10-10T01:43:40.0910811Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:40.0911468Z 2025-10-10T01:43:40.0911977Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:40.0912534Z 2025-10-10T01:43:40.0913135Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:40.0913658Z 2025-10-10T01:43:40.0914004Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:40.0914353Z 2025-10-10T01:43:40.0914817Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0915114Z 2025-10-10T01:43:40.0915614Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:40.0916066Z 2025-10-10T01:43:40.0916337Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:43:40.0916608Z 2025-10-10T01:43:40.0916855Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0917111Z 2025-10-10T01:43:40.0917627Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:40.0918105Z 2025-10-10T01:43:40.0918407Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:40.0918698Z 2025-10-10T01:43:40.0918963Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0919377Z 2025-10-10T01:43:40.0919933Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:40.0920433Z 2025-10-10T01:43:40.0920820Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:43:40.0921131Z 2025-10-10T01:43:40.0921416Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0921697Z 2025-10-10T01:43:40.0922149Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:40.0922567Z 2025-10-10T01:43:40.0922913Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:40.0923237Z 2025-10-10T01:43:40.0923536Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0923811Z 2025-10-10T01:43:40.0924152Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:43:40.0924493Z 2025-10-10T01:43:40.0924855Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:43:40.0925238Z 2025-10-10T01:43:40.0925636Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:40.0926001Z 2025-10-10T01:43:40.0926357Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:43:40.0926699Z 2025-10-10T01:43:40.0927059Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:43:40.0927411Z 2025-10-10T01:43:40.0927784Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:40.0928141Z 2025-10-10T01:43:40.0928641Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:40.0929110Z 2025-10-10T01:43:40.0929550Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:40.0929899Z 2025-10-10T01:43:40.0930447Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:40.0930924Z 2025-10-10T01:43:40.0931311Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:40.0931670Z 2025-10-10T01:43:40.0932218Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:40.0932725Z 2025-10-10T01:43:40.0933013Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:43:40.0933317Z 2025-10-10T01:43:40.0933942Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:40.0934514Z 2025-10-10T01:43:40.0934887Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:40.0935242Z 2025-10-10T01:43:40.0935495Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0935818Z 2025-10-10T01:43:40.0936398Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:40.0936913Z 2025-10-10T01:43:40.0937231Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:43:40.0937522Z 2025-10-10T01:43:40.0937799Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0938067Z 2025-10-10T01:43:40.0938677Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:40.0939226Z 2025-10-10T01:43:40.0939503Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:43:40.0939804Z 2025-10-10T01:43:40.0940072Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0940345Z 2025-10-10T01:43:40.0940991Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:40.0941528Z 2025-10-10T01:43:40.0941859Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:40.0942164Z 2025-10-10T01:43:40.0942450Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0942721Z 2025-10-10T01:43:40.0943255Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:40.0943738Z 2025-10-10T01:43:40.0944023Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:43:40.0944308Z 2025-10-10T01:43:40.0944682Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0944956Z 2025-10-10T01:43:40.0945491Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:40.0945970Z 2025-10-10T01:43:40.0946292Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:43:40.0946603Z 2025-10-10T01:43:40.0946838Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:43:40.0947093Z 2025-10-10T01:43:40.0947580Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:40.0948027Z 2025-10-10T01:43:40.0948277Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] raise RuntimeError( 2025-10-10T01:43:40.0948536Z 2025-10-10T01:43:40.0949109Z (EngineCore_DP0 pid=4121) ERROR 10-10 01:43:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:40.0949833Z (EngineCore_DP0 pid=4121) Process EngineCore_DP0: 2025-10-10T01:43:40.0950234Z (EngineCore_DP0 pid=4121) Traceback (most recent call last): 2025-10-10T01:43:40.0950846Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:43:40.0951425Z (EngineCore_DP0 pid=4121) self.run() 2025-10-10T01:43:40.0951947Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:43:40.0952514Z (EngineCore_DP0 pid=4121) self._target(*self._args, **self._kwargs) 2025-10-10T01:43:40.0953183Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:43:40.0953731Z (EngineCore_DP0 pid=4121) raise e 2025-10-10T01:43:40.0954338Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:40.0954984Z (EngineCore_DP0 pid=4121) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:40.0955435Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0956057Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:40.0956712Z (EngineCore_DP0 pid=4121) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:40.0957434Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:40.0958056Z (EngineCore_DP0 pid=4121) self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:40.0958514Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0959229Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:40.0959821Z (EngineCore_DP0 pid=4121) self._init_executor() 2025-10-10T01:43:40.0960486Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:40.0961169Z (EngineCore_DP0 pid=4121) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:40.0961983Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:40.0962712Z (EngineCore_DP0 pid=4121) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:40.0963213Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0963857Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:40.0964448Z (EngineCore_DP0 pid=4121) return func(*args, **kwargs) 2025-10-10T01:43:40.0964833Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0965470Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:40.0966111Z (EngineCore_DP0 pid=4121) worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:40.0966549Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0967225Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:40.0967893Z (EngineCore_DP0 pid=4121) module = importlib.import_module(module_name) 2025-10-10T01:43:40.0968339Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0968983Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:40.0969616Z (EngineCore_DP0 pid=4121) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:40.0970118Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0970621Z (EngineCore_DP0 pid=4121) File "", line 1387, in _gcd_import 2025-10-10T01:43:40.0971174Z (EngineCore_DP0 pid=4121) File "", line 1360, in _find_and_load 2025-10-10T01:43:40.0971760Z (EngineCore_DP0 pid=4121) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:40.0972338Z (EngineCore_DP0 pid=4121) File "", line 935, in _load_unlocked 2025-10-10T01:43:40.0972921Z (EngineCore_DP0 pid=4121) File "", line 999, in exec_module 2025-10-10T01:43:40.0973535Z (EngineCore_DP0 pid=4121) File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:40.0974271Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:40.0975015Z (EngineCore_DP0 pid=4121) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:40.0975753Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:40.0976471Z (EngineCore_DP0 pid=4121) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:40.0977253Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:40.0977920Z (EngineCore_DP0 pid=4121) class FlashAttentionMetadataBuilder( 2025-10-10T01:43:40.0978724Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:40.0979623Z (EngineCore_DP0 pid=4121) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:40.0980115Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0980809Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:40.0981505Z (EngineCore_DP0 pid=4121) if not is_fa_version_supported(fa_version): 2025-10-10T01:43:40.0981950Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0982705Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:40.0983397Z (EngineCore_DP0 pid=4121) return _is_fa2_supported(device)[0] 2025-10-10T01:43:40.0983800Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0984511Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:40.0985222Z (EngineCore_DP0 pid=4121) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:40.0985671Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0986339Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:40.0987035Z (EngineCore_DP0 pid=4121) prop = get_device_properties(device) 2025-10-10T01:43:40.0987464Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:40.0988135Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:40.0988809Z (EngineCore_DP0 pid=4121) _lazy_init() # will define _get_device_properties 2025-10-10T01:43:40.0989212Z (EngineCore_DP0 pid=4121) ^^^^^^^^^^^^ 2025-10-10T01:43:40.0989785Z (EngineCore_DP0 pid=4121) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:40.0990344Z (EngineCore_DP0 pid=4121) raise RuntimeError( 2025-10-10T01:43:40.0991021Z (EngineCore_DP0 pid=4121) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:40.4960659Z FAILED 2025-10-10T01:43:40.5087954Z models/test_initialization.py::test_can_initialize_large_subset[LlavaNextVideoForConditionalGeneration] Fork a new process to run a test 4125 2025-10-10T01:43:40.5098782Z Fork a new process to run a test 0 2025-10-10T01:43:40.5369757Z INFO 10-10 01:43:40 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlavaNextVideoForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'llava-hf/LLaVA-NeXT-Video-7B-hf'} 2025-10-10T01:43:40.6104204Z 2025-10-10T01:43:40.6106272Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:43:40.6106596Z config.json: 1.41kB [00:00, 8.27MB/s] 2025-10-10T01:43:40.7252211Z 2025-10-10T01:43:40.7252924Z preprocessor_config.json: 0% 0.00/741 [00:00", line 1387, in _gcd_import 2025-10-10T01:43:48.8966947Z 2025-10-10T01:43:48.8967198Z ERROR 10-10 01:43:48 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:43:48.8967545Z 2025-10-10T01:43:48.8967770Z ERROR 10-10 01:43:48 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:48.8968079Z 2025-10-10T01:43:48.8968275Z ERROR 10-10 01:43:48 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:43:48.8968546Z 2025-10-10T01:43:48.8968766Z ERROR 10-10 01:43:48 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:43:48.8969070Z 2025-10-10T01:43:48.8969443Z ERROR 10-10 01:43:48 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:48.8969762Z 2025-10-10T01:43:48.8970150Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava_next_video.py", line 30, in 2025-10-10T01:43:48.8970619Z 2025-10-10T01:43:48.8970793Z ERROR 10-10 01:43:48 [registry.py:542] from .llava import init_vision_tower_for_llava 2025-10-10T01:43:48.8971034Z 2025-10-10T01:43:48.8971377Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava.py", line 40, in 2025-10-10T01:43:48.8971811Z 2025-10-10T01:43:48.8972023Z ERROR 10-10 01:43:48 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T01:43:48.8972343Z 2025-10-10T01:43:48.8972919Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:43:48.8973567Z 2025-10-10T01:43:48.8973722Z ERROR 10-10 01:43:48 [registry.py:542] from xformers import ops as xops 2025-10-10T01:43:48.8973946Z 2025-10-10T01:43:48.8974253Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:43:48.8974646Z 2025-10-10T01:43:48.8974761Z ERROR 10-10 01:43:48 [registry.py:542] from .fmha import ( 2025-10-10T01:43:48.8975030Z 2025-10-10T01:43:48.8975360Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:43:48.8975758Z 2025-10-10T01:43:48.8975995Z ERROR 10-10 01:43:48 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:43:48.8976295Z 2025-10-10T01:43:48.8976659Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:43:48.8977083Z 2025-10-10T01:43:48.8977248Z ERROR 10-10 01:43:48 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:43:48.8977487Z 2025-10-10T01:43:48.8977621Z ERROR 10-10 01:43:48 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:48.8977855Z 2025-10-10T01:43:48.8978180Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:43:48.8978574Z 2025-10-10T01:43:48.8978684Z ERROR 10-10 01:43:48 [registry.py:542] value = func() 2025-10-10T01:43:48.8978864Z 2025-10-10T01:43:48.8978965Z ERROR 10-10 01:43:48 [registry.py:542] ^^^^^^ 2025-10-10T01:43:48.8979135Z 2025-10-10T01:43:48.8979553Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:43:48.8979970Z 2025-10-10T01:43:48.8980162Z ERROR 10-10 01:43:48 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:43:48.8980419Z 2025-10-10T01:43:48.8980554Z ERROR 10-10 01:43:48 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:48.8980773Z 2025-10-10T01:43:48.8981130Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:48.8981559Z 2025-10-10T01:43:48.8981707Z ERROR 10-10 01:43:48 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:43:48.8981937Z 2025-10-10T01:43:48.8982065Z ERROR 10-10 01:43:48 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:48.8982277Z 2025-10-10T01:43:48.8982718Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:48.8983144Z 2025-10-10T01:43:48.8983318Z ERROR 10-10 01:43:48 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:43:48.8983562Z 2025-10-10T01:43:48.8983666Z ERROR 10-10 01:43:48 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:43:48.8983843Z 2025-10-10T01:43:48.8984164Z ERROR 10-10 01:43:48 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:48.8984559Z 2025-10-10T01:43:48.8984677Z ERROR 10-10 01:43:48 [registry.py:542] raise RuntimeError( 2025-10-10T01:43:48.8984866Z 2025-10-10T01:43:48.8985265Z ERROR 10-10 01:43:48 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:49.2717266Z FAILED 2025-10-10T01:43:49.2849954Z models/test_initialization.py::test_can_initialize_large_subset[GemmaForCausalLM] Fork a new process to run a test 4201 2025-10-10T01:43:49.2861801Z Fork a new process to run a test 0 2025-10-10T01:43:49.3132472Z INFO 10-10 01:43:49 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GemmaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/gemma-1.1-2b-it'} 2025-10-10T01:43:49.4390499Z 2025-10-10T01:43:49.4391430Z config.json: 0% 0.00/618 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:43:58.4442211Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] EngineCore failed to start. 2025-10-10T01:43:58.4443081Z 2025-10-10T01:43:58.4443518Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] Traceback (most recent call last): 2025-10-10T01:43:58.4443879Z 2025-10-10T01:43:58.4444567Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:58.4445180Z 2025-10-10T01:43:58.4445565Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:58.4445963Z 2025-10-10T01:43:58.4446272Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4446553Z 2025-10-10T01:43:58.4447034Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:58.4447483Z 2025-10-10T01:43:58.4447812Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:58.4448140Z 2025-10-10T01:43:58.4448789Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:58.4449416Z 2025-10-10T01:43:58.4449760Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:58.4450065Z 2025-10-10T01:43:58.4450349Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4450630Z 2025-10-10T01:43:58.4451139Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:58.4451616Z 2025-10-10T01:43:58.4451865Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] self._init_executor() 2025-10-10T01:43:58.4452127Z 2025-10-10T01:43:58.4452913Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:58.4453807Z 2025-10-10T01:43:58.4454476Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:58.4455059Z 2025-10-10T01:43:58.4456107Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:58.4456846Z 2025-10-10T01:43:58.4457234Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:58.4457747Z 2025-10-10T01:43:58.4458057Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4458371Z 2025-10-10T01:43:58.4458900Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:58.4459368Z 2025-10-10T01:43:58.4459643Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:43:58.4459920Z 2025-10-10T01:43:58.4460182Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4460445Z 2025-10-10T01:43:58.4460974Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:58.4461526Z 2025-10-10T01:43:58.4461818Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:58.4462113Z 2025-10-10T01:43:58.4462381Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4462654Z 2025-10-10T01:43:58.4463179Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:58.4463671Z 2025-10-10T01:43:58.4463974Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:43:58.4464279Z 2025-10-10T01:43:58.4464553Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4464824Z 2025-10-10T01:43:58.4465273Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:58.4465750Z 2025-10-10T01:43:58.4466102Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:58.4466430Z 2025-10-10T01:43:58.4466712Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4466991Z 2025-10-10T01:43:58.4467332Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:43:58.4467661Z 2025-10-10T01:43:58.4468007Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:43:58.4468348Z 2025-10-10T01:43:58.4468723Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:58.4469084Z 2025-10-10T01:43:58.4469504Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:43:58.4469838Z 2025-10-10T01:43:58.4470204Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:43:58.4470552Z 2025-10-10T01:43:58.4470923Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:58.4471278Z 2025-10-10T01:43:58.4471777Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:58.4472239Z 2025-10-10T01:43:58.4472596Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:58.4472940Z 2025-10-10T01:43:58.4473463Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:58.4473946Z 2025-10-10T01:43:58.4474315Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:58.4474674Z 2025-10-10T01:43:58.4475227Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:58.4475780Z 2025-10-10T01:43:58.4476070Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:43:58.4476369Z 2025-10-10T01:43:58.4476998Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:58.4477568Z 2025-10-10T01:43:58.4477936Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:58.4478288Z 2025-10-10T01:43:58.4478546Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4478803Z 2025-10-10T01:43:58.4479497Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:58.4480012Z 2025-10-10T01:43:58.4480311Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:43:58.4480609Z 2025-10-10T01:43:58.4480956Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4481237Z 2025-10-10T01:43:58.4481984Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:58.4482543Z 2025-10-10T01:43:58.4482824Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:43:58.4483111Z 2025-10-10T01:43:58.4483369Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4483631Z 2025-10-10T01:43:58.4484214Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:58.4484741Z 2025-10-10T01:43:58.4485193Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:58.4485506Z 2025-10-10T01:43:58.4485793Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4486063Z 2025-10-10T01:43:58.4486584Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:58.4487071Z 2025-10-10T01:43:58.4487347Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:43:58.4487637Z 2025-10-10T01:43:58.4487895Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4488169Z 2025-10-10T01:43:58.4488704Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:58.4489187Z 2025-10-10T01:43:58.4489497Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:43:58.4489800Z 2025-10-10T01:43:58.4490034Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:43:58.4490275Z 2025-10-10T01:43:58.4490759Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:58.4491253Z 2025-10-10T01:43:58.4491505Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] raise RuntimeError( 2025-10-10T01:43:58.4491767Z 2025-10-10T01:43:58.4492338Z (EngineCore_DP0 pid=4281) ERROR 10-10 01:43:58 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:58.4493030Z (EngineCore_DP0 pid=4281) Process EngineCore_DP0: 2025-10-10T01:43:58.4493424Z (EngineCore_DP0 pid=4281) Traceback (most recent call last): 2025-10-10T01:43:58.4494029Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:43:58.4494573Z (EngineCore_DP0 pid=4281) self.run() 2025-10-10T01:43:58.4495102Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:43:58.4495669Z (EngineCore_DP0 pid=4281) self._target(*self._args, **self._kwargs) 2025-10-10T01:43:58.4496736Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:43:58.4497447Z (EngineCore_DP0 pid=4281) raise e 2025-10-10T01:43:58.4498049Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:43:58.4498682Z (EngineCore_DP0 pid=4281) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:43:58.4499119Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4499733Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:43:58.4500373Z (EngineCore_DP0 pid=4281) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:43:58.4501028Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:43:58.4501645Z (EngineCore_DP0 pid=4281) self.model_executor = executor_class(vllm_config) 2025-10-10T01:43:58.4502211Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4502862Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:43:58.4503448Z (EngineCore_DP0 pid=4281) self._init_executor() 2025-10-10T01:43:58.4504140Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:43:58.4504830Z (EngineCore_DP0 pid=4281) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:43:58.4505551Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:43:58.4506260Z (EngineCore_DP0 pid=4281) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:43:58.4506754Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4507395Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:43:58.4507990Z (EngineCore_DP0 pid=4281) return func(*args, **kwargs) 2025-10-10T01:43:58.4508373Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4509080Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:43:58.4509707Z (EngineCore_DP0 pid=4281) worker_class = resolve_obj_by_qualname( 2025-10-10T01:43:58.4510131Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4510809Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:43:58.4511481Z (EngineCore_DP0 pid=4281) module = importlib.import_module(module_name) 2025-10-10T01:43:58.4511930Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4512510Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:43:58.4513138Z (EngineCore_DP0 pid=4281) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:43:58.4513631Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4514127Z (EngineCore_DP0 pid=4281) File "", line 1387, in _gcd_import 2025-10-10T01:43:58.4514730Z (EngineCore_DP0 pid=4281) File "", line 1360, in _find_and_load 2025-10-10T01:43:58.4515318Z (EngineCore_DP0 pid=4281) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:43:58.4515897Z (EngineCore_DP0 pid=4281) File "", line 935, in _load_unlocked 2025-10-10T01:43:58.4516470Z (EngineCore_DP0 pid=4281) File "", line 999, in exec_module 2025-10-10T01:43:58.4517068Z (EngineCore_DP0 pid=4281) File "", line 488, in _call_with_frames_removed 2025-10-10T01:43:58.4517814Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:43:58.4518496Z (EngineCore_DP0 pid=4281) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:43:58.4519406Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:43:58.4520144Z (EngineCore_DP0 pid=4281) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:43:58.4520928Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:43:58.4521606Z (EngineCore_DP0 pid=4281) class FlashAttentionMetadataBuilder( 2025-10-10T01:43:58.4522402Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:43:58.4523209Z (EngineCore_DP0 pid=4281) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:43:58.4523697Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4524390Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:43:58.4525072Z (EngineCore_DP0 pid=4281) if not is_fa_version_supported(fa_version): 2025-10-10T01:43:58.4525507Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4526252Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:43:58.4527000Z (EngineCore_DP0 pid=4281) return _is_fa2_supported(device)[0] 2025-10-10T01:43:58.4527410Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4528119Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:43:58.4528839Z (EngineCore_DP0 pid=4281) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:43:58.4529283Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4529951Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:43:58.4530586Z (EngineCore_DP0 pid=4281) prop = get_device_properties(device) 2025-10-10T01:43:58.4530997Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:43:58.4531653Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:43:58.4532311Z (EngineCore_DP0 pid=4281) _lazy_init() # will define _get_device_properties 2025-10-10T01:43:58.4532749Z (EngineCore_DP0 pid=4281) ^^^^^^^^^^^^ 2025-10-10T01:43:58.4533332Z (EngineCore_DP0 pid=4281) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:43:58.4533891Z (EngineCore_DP0 pid=4281) raise RuntimeError( 2025-10-10T01:43:58.4534571Z (EngineCore_DP0 pid=4281) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:43:58.8613290Z FAILED 2025-10-10T01:43:58.8740345Z models/test_initialization.py::test_can_initialize_large_subset[MiniMaxVL01ForConditionalGeneration] Fork a new process to run a test 4285 2025-10-10T01:43:58.8751100Z Fork a new process to run a test 0 2025-10-10T01:43:59.1743016Z PASSED 2025-10-10T01:43:59.1870333Z models/test_initialization.py::test_can_initialize_large_subset[Gemma2Model] Fork a new process to run a test 4286 2025-10-10T01:43:59.1882271Z Fork a new process to run a test 0 2025-10-10T01:43:59.2158844Z INFO 10-10 01:43:59 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma2Model', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/bge-multilingual-gemma2'} 2025-10-10T01:43:59.2957267Z 2025-10-10T01:43:59.2957818Z config.json: 0% 0.00/897 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:44:08.1060637Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] EngineCore failed to start. 2025-10-10T01:44:08.1061184Z 2025-10-10T01:44:08.1061609Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] Traceback (most recent call last): 2025-10-10T01:44:08.1062161Z 2025-10-10T01:44:08.1062935Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:08.1063605Z 2025-10-10T01:44:08.1064155Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:08.1064988Z 2025-10-10T01:44:08.1065314Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1065639Z 2025-10-10T01:44:08.1066226Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:08.1066796Z 2025-10-10T01:44:08.1067155Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:08.1067605Z 2025-10-10T01:44:08.1068094Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:08.1068658Z 2025-10-10T01:44:08.1069023Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:08.1069473Z 2025-10-10T01:44:08.1069954Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1070391Z 2025-10-10T01:44:08.1070944Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:08.1071520Z 2025-10-10T01:44:08.1071793Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] self._init_executor() 2025-10-10T01:44:08.1072196Z 2025-10-10T01:44:08.1072976Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:08.1073584Z 2025-10-10T01:44:08.1074064Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:08.1074405Z 2025-10-10T01:44:08.1075103Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:08.1075729Z 2025-10-10T01:44:08.1076100Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:08.1076570Z 2025-10-10T01:44:08.1076874Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1077405Z 2025-10-10T01:44:08.1077950Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:08.1078554Z 2025-10-10T01:44:08.1078841Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:44:08.1079348Z 2025-10-10T01:44:08.1079627Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1079890Z 2025-10-10T01:44:08.1080555Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:08.1081166Z 2025-10-10T01:44:08.1081469Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:08.1081826Z 2025-10-10T01:44:08.1082190Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1082470Z 2025-10-10T01:44:08.1083143Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:08.1083832Z 2025-10-10T01:44:08.1084164Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:44:08.1084574Z 2025-10-10T01:44:08.1084896Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1085172Z 2025-10-10T01:44:08.1085750Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:08.1086202Z 2025-10-10T01:44:08.1086647Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:08.1086985Z 2025-10-10T01:44:08.1087428Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1087826Z 2025-10-10T01:44:08.1088336Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:44:08.1088691Z 2025-10-10T01:44:08.1089183Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:44:08.1089529Z 2025-10-10T01:44:08.1090043Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:08.1090427Z 2025-10-10T01:44:08.1090924Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:44:08.1091267Z 2025-10-10T01:44:08.1091770Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:44:08.1092131Z 2025-10-10T01:44:08.1092647Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:08.1093015Z 2025-10-10T01:44:08.1093668Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:08.1094187Z 2025-10-10T01:44:08.1094648Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:08.1095122Z 2025-10-10T01:44:08.1095756Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:08.1096539Z 2025-10-10T01:44:08.1097077Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:08.1097455Z 2025-10-10T01:44:08.1098156Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:08.1098793Z 2025-10-10T01:44:08.1099103Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:44:08.1099410Z 2025-10-10T01:44:08.1100181Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:08.1100896Z 2025-10-10T01:44:08.1101391Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:08.1101867Z 2025-10-10T01:44:08.1102250Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1102558Z 2025-10-10T01:44:08.1103267Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:08.1103803Z 2025-10-10T01:44:08.1104185Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:44:08.1104536Z 2025-10-10T01:44:08.1104825Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1105105Z 2025-10-10T01:44:08.1105826Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:08.1106566Z 2025-10-10T01:44:08.1106903Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:44:08.1107195Z 2025-10-10T01:44:08.1107464Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1107729Z 2025-10-10T01:44:08.1108490Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:08.1109037Z 2025-10-10T01:44:08.1109370Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:08.1109679Z 2025-10-10T01:44:08.1109955Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1110228Z 2025-10-10T01:44:08.1110768Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:08.1111262Z 2025-10-10T01:44:08.1111544Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:44:08.1111948Z 2025-10-10T01:44:08.1112240Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1112622Z 2025-10-10T01:44:08.1113170Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:08.1113669Z 2025-10-10T01:44:08.1113991Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:08.1114306Z 2025-10-10T01:44:08.1114538Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:44:08.1114779Z 2025-10-10T01:44:08.1115267Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:08.1115720Z 2025-10-10T01:44:08.1115961Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] raise RuntimeError( 2025-10-10T01:44:08.1116236Z 2025-10-10T01:44:08.1116799Z (EngineCore_DP0 pid=4365) ERROR 10-10 01:44:08 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:08.1117486Z (EngineCore_DP0 pid=4365) Process EngineCore_DP0: 2025-10-10T01:44:08.1117925Z (EngineCore_DP0 pid=4365) Traceback (most recent call last): 2025-10-10T01:44:08.1118552Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:44:08.1119282Z (EngineCore_DP0 pid=4365) self.run() 2025-10-10T01:44:08.1119852Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:44:08.1120417Z (EngineCore_DP0 pid=4365) self._target(*self._args, **self._kwargs) 2025-10-10T01:44:08.1121087Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:44:08.1121663Z (EngineCore_DP0 pid=4365) raise e 2025-10-10T01:44:08.1122258Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:08.1122984Z (EngineCore_DP0 pid=4365) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:08.1123430Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1124052Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:08.1124692Z (EngineCore_DP0 pid=4365) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:08.1125348Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:08.1125967Z (EngineCore_DP0 pid=4365) self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:08.1126414Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1127063Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:08.1127659Z (EngineCore_DP0 pid=4365) self._init_executor() 2025-10-10T01:44:08.1128346Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:08.1129038Z (EngineCore_DP0 pid=4365) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:08.1129761Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:08.1130511Z (EngineCore_DP0 pid=4365) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:08.1131004Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1131652Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:08.1132240Z (EngineCore_DP0 pid=4365) return func(*args, **kwargs) 2025-10-10T01:44:08.1132790Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1133442Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:08.1134079Z (EngineCore_DP0 pid=4365) worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:08.1134514Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1135182Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:08.1135899Z (EngineCore_DP0 pid=4365) module = importlib.import_module(module_name) 2025-10-10T01:44:08.1136356Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1136940Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:08.1137567Z (EngineCore_DP0 pid=4365) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:08.1138061Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1138566Z (EngineCore_DP0 pid=4365) File "", line 1387, in _gcd_import 2025-10-10T01:44:08.1139123Z (EngineCore_DP0 pid=4365) File "", line 1360, in _find_and_load 2025-10-10T01:44:08.1139711Z (EngineCore_DP0 pid=4365) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:08.1140364Z (EngineCore_DP0 pid=4365) File "", line 935, in _load_unlocked 2025-10-10T01:44:08.1140945Z (EngineCore_DP0 pid=4365) File "", line 999, in exec_module 2025-10-10T01:44:08.1141546Z (EngineCore_DP0 pid=4365) File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:08.1142287Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:08.1142976Z (EngineCore_DP0 pid=4365) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:08.1143700Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:08.1144420Z (EngineCore_DP0 pid=4365) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:08.1145213Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:08.1145881Z (EngineCore_DP0 pid=4365) class FlashAttentionMetadataBuilder( 2025-10-10T01:44:08.1146664Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:08.1147474Z (EngineCore_DP0 pid=4365) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:08.1148006Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1148696Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:08.1149383Z (EngineCore_DP0 pid=4365) if not is_fa_version_supported(fa_version): 2025-10-10T01:44:08.1149817Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1150557Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:08.1151252Z (EngineCore_DP0 pid=4365) return _is_fa2_supported(device)[0] 2025-10-10T01:44:08.1151662Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1152373Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:08.1153084Z (EngineCore_DP0 pid=4365) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:08.1153577Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1154261Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:08.1154896Z (EngineCore_DP0 pid=4365) prop = get_device_properties(device) 2025-10-10T01:44:08.1155309Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:08.1155966Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:08.1156633Z (EngineCore_DP0 pid=4365) _lazy_init() # will define _get_device_properties 2025-10-10T01:44:08.1157025Z (EngineCore_DP0 pid=4365) ^^^^^^^^^^^^ 2025-10-10T01:44:08.1157605Z (EngineCore_DP0 pid=4365) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:08.1158169Z (EngineCore_DP0 pid=4365) raise RuntimeError( 2025-10-10T01:44:08.1159133Z (EngineCore_DP0 pid=4365) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:08.5248261Z FAILED 2025-10-10T01:44:08.5375176Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5OmniModel] Fork a new process to run a test 4369 2025-10-10T01:44:08.5386389Z Fork a new process to run a test 0 2025-10-10T01:44:08.5668201Z INFO 10-10 01:44:08 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2_5OmniModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen2.5-Omni-3B'} 2025-10-10T01:44:08.6495606Z 2025-10-10T01:44:08.6499164Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:08.6499479Z config.json: 13.2kB [00:00, 41.4MB/s] 2025-10-10T01:44:08.6891677Z Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'} 2025-10-10T01:44:08.7569344Z 2025-10-10T01:44:08.7570727Z preprocessor_config.json: 0% 0.00/667 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:44:17.3915789Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] EngineCore failed to start. 2025-10-10T01:44:17.3916241Z 2025-10-10T01:44:17.3916764Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] Traceback (most recent call last): 2025-10-10T01:44:17.3918184Z 2025-10-10T01:44:17.3919160Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:17.3919957Z 2025-10-10T01:44:17.3920491Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:17.3920913Z 2025-10-10T01:44:17.3921371Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3921777Z 2025-10-10T01:44:17.3922535Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:17.3923180Z 2025-10-10T01:44:17.3923704Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:17.3931469Z 2025-10-10T01:44:17.3932148Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:17.3932818Z 2025-10-10T01:44:17.3933325Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:17.3933674Z 2025-10-10T01:44:17.3934135Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3934421Z 2025-10-10T01:44:17.3935077Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:17.3935574Z 2025-10-10T01:44:17.3935969Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] self._init_executor() 2025-10-10T01:44:17.3936240Z 2025-10-10T01:44:17.3936921Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:17.3937549Z 2025-10-10T01:44:17.3938078Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:17.3938571Z 2025-10-10T01:44:17.3939164Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:17.3939778Z 2025-10-10T01:44:17.3940248Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:17.3940607Z 2025-10-10T01:44:17.3940940Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3941322Z 2025-10-10T01:44:17.3941894Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:17.3942415Z 2025-10-10T01:44:17.3942753Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:44:17.3943102Z 2025-10-10T01:44:17.3943372Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3943736Z 2025-10-10T01:44:17.3944293Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:17.3944978Z 2025-10-10T01:44:17.3945294Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:17.3945717Z 2025-10-10T01:44:17.3945996Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3946382Z 2025-10-10T01:44:17.3946949Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:17.3947569Z 2025-10-10T01:44:17.3947885Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:44:17.3948315Z 2025-10-10T01:44:17.3948607Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3948958Z 2025-10-10T01:44:17.3949471Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:17.3950022Z 2025-10-10T01:44:17.3950393Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:17.3950930Z 2025-10-10T01:44:17.3951262Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3951646Z 2025-10-10T01:44:17.3952026Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:44:17.3952408Z 2025-10-10T01:44:17.3952850Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:44:17.3953206Z 2025-10-10T01:44:17.3953714Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:17.3954100Z 2025-10-10T01:44:17.3954568Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:44:17.3954910Z 2025-10-10T01:44:17.3955503Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:44:17.3955888Z 2025-10-10T01:44:17.3956392Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:17.3956781Z 2025-10-10T01:44:17.3957430Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:17.3958040Z 2025-10-10T01:44:17.3958421Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:17.3958832Z 2025-10-10T01:44:17.3959509Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:17.3960005Z 2025-10-10T01:44:17.3960523Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:17.3960900Z 2025-10-10T01:44:17.3961474Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:17.3961987Z 2025-10-10T01:44:17.3962280Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:44:17.3962809Z 2025-10-10T01:44:17.3963479Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:17.3964078Z 2025-10-10T01:44:17.3964462Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:17.3964817Z 2025-10-10T01:44:17.3965078Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3965343Z 2025-10-10T01:44:17.3965913Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:17.3966461Z 2025-10-10T01:44:17.3966888Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:44:17.3967193Z 2025-10-10T01:44:17.3967471Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3967835Z 2025-10-10T01:44:17.3968448Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:17.3968998Z 2025-10-10T01:44:17.3969277Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:44:17.3969565Z 2025-10-10T01:44:17.3969823Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3970087Z 2025-10-10T01:44:17.3970668Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:17.3971197Z 2025-10-10T01:44:17.3971525Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:17.3971838Z 2025-10-10T01:44:17.3972198Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3972477Z 2025-10-10T01:44:17.3973000Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:17.3973488Z 2025-10-10T01:44:17.3973927Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:44:17.3974226Z 2025-10-10T01:44:17.3974497Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3974774Z 2025-10-10T01:44:17.3975289Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:17.3975775Z 2025-10-10T01:44:17.3976089Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:17.3976391Z 2025-10-10T01:44:17.3976624Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:44:17.3976865Z 2025-10-10T01:44:17.3977352Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:17.3977795Z 2025-10-10T01:44:17.3978048Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] raise RuntimeError( 2025-10-10T01:44:17.3978368Z 2025-10-10T01:44:17.3978935Z (EngineCore_DP0 pid=4448) ERROR 10-10 01:44:17 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:17.3979633Z (EngineCore_DP0 pid=4448) Process EngineCore_DP0: 2025-10-10T01:44:17.3980031Z (EngineCore_DP0 pid=4448) Traceback (most recent call last): 2025-10-10T01:44:17.3980643Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:44:17.3981159Z (EngineCore_DP0 pid=4448) self.run() 2025-10-10T01:44:17.3981688Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:44:17.3982283Z (EngineCore_DP0 pid=4448) self._target(*self._args, **self._kwargs) 2025-10-10T01:44:17.3982964Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:44:17.3983525Z (EngineCore_DP0 pid=4448) raise e 2025-10-10T01:44:17.3984192Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:17.3984827Z (EngineCore_DP0 pid=4448) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:17.3985271Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3985895Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:17.3986537Z (EngineCore_DP0 pid=4448) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:17.3987345Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:17.3987987Z (EngineCore_DP0 pid=4448) self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:17.3988474Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3989242Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:17.3989843Z (EngineCore_DP0 pid=4448) self._init_executor() 2025-10-10T01:44:17.3990506Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:17.3991195Z (EngineCore_DP0 pid=4448) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:17.3991914Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:17.3992631Z (EngineCore_DP0 pid=4448) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:17.3993130Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3993793Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:17.3994391Z (EngineCore_DP0 pid=4448) return func(*args, **kwargs) 2025-10-10T01:44:17.3994771Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3995430Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:17.3996355Z (EngineCore_DP0 pid=4448) worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:17.3996799Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3997473Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:17.3998393Z (EngineCore_DP0 pid=4448) module = importlib.import_module(module_name) 2025-10-10T01:44:17.3998858Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.3999533Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:17.4000164Z (EngineCore_DP0 pid=4448) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:17.4000659Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.4001152Z (EngineCore_DP0 pid=4448) File "", line 1387, in _gcd_import 2025-10-10T01:44:17.4001717Z (EngineCore_DP0 pid=4448) File "", line 1360, in _find_and_load 2025-10-10T01:44:17.4002441Z (EngineCore_DP0 pid=4448) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:17.4003026Z (EngineCore_DP0 pid=4448) File "", line 935, in _load_unlocked 2025-10-10T01:44:17.4003603Z (EngineCore_DP0 pid=4448) File "", line 999, in exec_module 2025-10-10T01:44:17.4004223Z (EngineCore_DP0 pid=4448) File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:17.4004985Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:17.4005681Z (EngineCore_DP0 pid=4448) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:17.4006410Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:17.4007261Z (EngineCore_DP0 pid=4448) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:17.4008062Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:17.4008728Z (EngineCore_DP0 pid=4448) class FlashAttentionMetadataBuilder( 2025-10-10T01:44:17.4009522Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:17.4010342Z (EngineCore_DP0 pid=4448) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:17.4010822Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.4011507Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:17.4012205Z (EngineCore_DP0 pid=4448) if not is_fa_version_supported(fa_version): 2025-10-10T01:44:17.4012646Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.4013438Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:17.4014303Z (EngineCore_DP0 pid=4448) return _is_fa2_supported(device)[0] 2025-10-10T01:44:17.4014826Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.4015532Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:17.4016248Z (EngineCore_DP0 pid=4448) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:17.4016703Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.4017383Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:17.4018022Z (EngineCore_DP0 pid=4448) prop = get_device_properties(device) 2025-10-10T01:44:17.4018434Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:17.4019089Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:17.4019749Z (EngineCore_DP0 pid=4448) _lazy_init() # will define _get_device_properties 2025-10-10T01:44:17.4020146Z (EngineCore_DP0 pid=4448) ^^^^^^^^^^^^ 2025-10-10T01:44:17.4020774Z (EngineCore_DP0 pid=4448) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:17.4021333Z (EngineCore_DP0 pid=4448) raise RuntimeError( 2025-10-10T01:44:17.4022015Z (EngineCore_DP0 pid=4448) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:17.8132710Z FAILED 2025-10-10T01:44:17.8259147Z models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeHybridForCausalLM] Fork a new process to run a test 4452 2025-10-10T01:44:17.8270411Z Fork a new process to run a test 0 2025-10-10T01:44:17.8549105Z INFO 10-10 01:44:17 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GraniteMoeHybridForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ibm-granite/granite-4.0-tiny-preview'} 2025-10-10T01:44:17.9744446Z 2025-10-10T01:44:17.9746827Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:17.9747185Z config.json: 1.80kB [00:00, 11.6MB/s] 2025-10-10T01:44:24.7631140Z INFO 10-10 01:44:24 [model.py:551] Resolved architecture: GraniteMoeHybridForCausalLM 2025-10-10T01:44:24.7631720Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:44:24.8721550Z 2025-10-10T01:44:24.8724717Z model.safetensors.index.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:24.8725150Z model.safetensors.index.json: 48.9kB [00:00, 111MB/s] 2025-10-10T01:44:24.8763371Z 2025-10-10T01:44:24.9348509Z Parse safetensors files: 0% 0/3 [00:00= mamba page size. 2025-10-10T01:44:25.1869796Z INFO 10-10 01:44:25 [config.py:397] Padding mamba page size by 1.59% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:44:25.2272185Z 2025-10-10T01:44:25.2274227Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:25.2274577Z tokenizer_config.json: 10.1kB [00:00, 53.4MB/s] 2025-10-10T01:44:25.3611460Z 2025-10-10T01:44:25.3759211Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:25.3759556Z tokenizer.json: 3.48MB [00:00, 233MB/s] 2025-10-10T01:44:25.4403521Z 2025-10-10T01:44:25.4404151Z special_tokens_map.json: 0% 0.00/801 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:44:25.8010137Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] EngineCore failed to start. 2025-10-10T01:44:25.8010738Z 2025-10-10T01:44:25.8011142Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] Traceback (most recent call last): 2025-10-10T01:44:25.8011529Z 2025-10-10T01:44:25.8012451Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:25.8013051Z 2025-10-10T01:44:25.8013455Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:25.8013834Z 2025-10-10T01:44:25.8014177Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8014718Z 2025-10-10T01:44:25.8015329Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:25.8015886Z 2025-10-10T01:44:25.8016303Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:25.8016711Z 2025-10-10T01:44:25.8017314Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:25.8017867Z 2025-10-10T01:44:25.8018253Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:25.8018628Z 2025-10-10T01:44:25.8019244Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8019602Z 2025-10-10T01:44:25.8020250Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:25.8020830Z 2025-10-10T01:44:25.8021239Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] self._init_executor() 2025-10-10T01:44:25.8021501Z 2025-10-10T01:44:25.8022042Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:25.8022542Z 2025-10-10T01:44:25.8022970Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:25.8023547Z 2025-10-10T01:44:25.8024440Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:25.8025056Z 2025-10-10T01:44:25.8025699Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:25.8026262Z 2025-10-10T01:44:25.8027001Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8027544Z 2025-10-10T01:44:25.8028212Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:25.8028679Z 2025-10-10T01:44:25.8028968Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:44:25.8029259Z 2025-10-10T01:44:25.8029523Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8029788Z 2025-10-10T01:44:25.8030322Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:25.8030810Z 2025-10-10T01:44:25.8031099Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:25.8031389Z 2025-10-10T01:44:25.8031652Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8031923Z 2025-10-10T01:44:25.8032447Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:25.8032933Z 2025-10-10T01:44:25.8033231Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:44:25.8033602Z 2025-10-10T01:44:25.8033907Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8034182Z 2025-10-10T01:44:25.8034626Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:25.8035053Z 2025-10-10T01:44:25.8035391Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:25.8035726Z 2025-10-10T01:44:25.8036008Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8036288Z 2025-10-10T01:44:25.8036625Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:44:25.8036952Z 2025-10-10T01:44:25.8037300Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:44:25.8037642Z 2025-10-10T01:44:25.8038091Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:25.8038453Z 2025-10-10T01:44:25.8038801Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:44:25.8039248Z 2025-10-10T01:44:25.8039611Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:44:25.8039965Z 2025-10-10T01:44:25.8040335Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:25.8040709Z 2025-10-10T01:44:25.8041210Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:25.8041684Z 2025-10-10T01:44:25.8042118Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:25.8042459Z 2025-10-10T01:44:25.8042984Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:25.8043457Z 2025-10-10T01:44:25.8043830Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:25.8044184Z 2025-10-10T01:44:25.8044740Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:25.8045240Z 2025-10-10T01:44:25.8045533Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:44:25.8045827Z 2025-10-10T01:44:25.8046449Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:25.8047026Z 2025-10-10T01:44:25.8047387Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:25.8047745Z 2025-10-10T01:44:25.8048000Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8048309Z 2025-10-10T01:44:25.8048867Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:25.8049383Z 2025-10-10T01:44:25.8049694Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:44:25.8049990Z 2025-10-10T01:44:25.8050266Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8050532Z 2025-10-10T01:44:25.8051135Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:25.8051679Z 2025-10-10T01:44:25.8051965Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:44:25.8052249Z 2025-10-10T01:44:25.8052509Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8052777Z 2025-10-10T01:44:25.8053354Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:25.8053942Z 2025-10-10T01:44:25.8054260Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:25.8054576Z 2025-10-10T01:44:25.8054845Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8055122Z 2025-10-10T01:44:25.8055641Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:25.8056130Z 2025-10-10T01:44:25.8056407Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:44:25.8056696Z 2025-10-10T01:44:25.8056961Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8057302Z 2025-10-10T01:44:25.8057842Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:25.8058333Z 2025-10-10T01:44:25.8058649Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:25.8058947Z 2025-10-10T01:44:25.8059173Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:44:25.8059417Z 2025-10-10T01:44:25.8059895Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:25.8060345Z 2025-10-10T01:44:25.8060586Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] raise RuntimeError( 2025-10-10T01:44:25.8060843Z 2025-10-10T01:44:25.8061406Z (EngineCore_DP0 pid=4513) ERROR 10-10 01:44:25 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:25.8062096Z (EngineCore_DP0 pid=4513) Process EngineCore_DP0: 2025-10-10T01:44:25.8062491Z (EngineCore_DP0 pid=4513) Traceback (most recent call last): 2025-10-10T01:44:25.8063092Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:44:25.8063662Z (EngineCore_DP0 pid=4513) self.run() 2025-10-10T01:44:25.8064214Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:44:25.8064781Z (EngineCore_DP0 pid=4513) self._target(*self._args, **self._kwargs) 2025-10-10T01:44:25.8065468Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:44:25.8066009Z (EngineCore_DP0 pid=4513) raise e 2025-10-10T01:44:25.8066595Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:25.8067225Z (EngineCore_DP0 pid=4513) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:25.8067676Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8068316Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:25.8068951Z (EngineCore_DP0 pid=4513) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:25.8069688Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:25.8070314Z (EngineCore_DP0 pid=4513) self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:25.8070768Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8071420Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:25.8071999Z (EngineCore_DP0 pid=4513) self._init_executor() 2025-10-10T01:44:25.8072661Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:25.8073346Z (EngineCore_DP0 pid=4513) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:25.8074142Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:25.8074875Z (EngineCore_DP0 pid=4513) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:25.8075370Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8076024Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:25.8076610Z (EngineCore_DP0 pid=4513) return func(*args, **kwargs) 2025-10-10T01:44:25.8076996Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8077630Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:25.8078260Z (EngineCore_DP0 pid=4513) worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:25.8078680Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8079414Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:25.8080068Z (EngineCore_DP0 pid=4513) module = importlib.import_module(module_name) 2025-10-10T01:44:25.8080518Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8081160Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:25.8081780Z (EngineCore_DP0 pid=4513) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:25.8082282Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8082782Z (EngineCore_DP0 pid=4513) File "", line 1387, in _gcd_import 2025-10-10T01:44:25.8083335Z (EngineCore_DP0 pid=4513) File "", line 1360, in _find_and_load 2025-10-10T01:44:25.8083923Z (EngineCore_DP0 pid=4513) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:25.8084492Z (EngineCore_DP0 pid=4513) File "", line 935, in _load_unlocked 2025-10-10T01:44:25.8085060Z (EngineCore_DP0 pid=4513) File "", line 999, in exec_module 2025-10-10T01:44:25.8085662Z (EngineCore_DP0 pid=4513) File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:25.8086389Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:25.8087131Z (EngineCore_DP0 pid=4513) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:25.8087843Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:25.8088563Z (EngineCore_DP0 pid=4513) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:25.8089338Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:25.8090004Z (EngineCore_DP0 pid=4513) class FlashAttentionMetadataBuilder( 2025-10-10T01:44:25.8090787Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:25.8091678Z (EngineCore_DP0 pid=4513) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:25.8092153Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8092845Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:25.8093524Z (EngineCore_DP0 pid=4513) if not is_fa_version_supported(fa_version): 2025-10-10T01:44:25.8093961Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8094708Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:25.8095408Z (EngineCore_DP0 pid=4513) return _is_fa2_supported(device)[0] 2025-10-10T01:44:25.8095821Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8096820Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:25.8097550Z (EngineCore_DP0 pid=4513) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:25.8098009Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8098681Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:25.8099424Z (EngineCore_DP0 pid=4513) prop = get_device_properties(device) 2025-10-10T01:44:25.8099839Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:25.8100502Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:25.8101171Z (EngineCore_DP0 pid=4513) _lazy_init() # will define _get_device_properties 2025-10-10T01:44:25.8101564Z (EngineCore_DP0 pid=4513) ^^^^^^^^^^^^ 2025-10-10T01:44:25.8102143Z (EngineCore_DP0 pid=4513) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:25.8102698Z (EngineCore_DP0 pid=4513) raise RuntimeError( 2025-10-10T01:44:25.8103379Z (EngineCore_DP0 pid=4513) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:26.2030792Z FAILED 2025-10-10T01:44:26.2157904Z models/test_initialization.py::test_can_initialize_large_subset[Llama_Nemotron_Nano_VL] Fork a new process to run a test 4517 2025-10-10T01:44:26.2169185Z Fork a new process to run a test 0 2025-10-10T01:44:26.2451857Z INFO 10-10 01:44:26 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Llama_Nemotron_Nano_VL', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1'} 2025-10-10T01:44:26.3494321Z 2025-10-10T01:44:26.5289398Z config.json: 0% 0.00/7.65k [00:00, model_arch='XverseForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'xverse/XVERSE-7B-Chat'} 2025-10-10T01:44:28.7776852Z 2025-10-10T01:44:28.7779112Z config.json: 0% 0.00/687 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message 2025-10-10T01:44:29.6669986Z You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. 2025-10-10T01:44:29.9590720Z 2025-10-10T01:44:29.9591324Z generation_config.json: 0% 0.00/232 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:44:30.1146735Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:44:30.1147146Z 2025-10-10T01:44:30.1147549Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:44:30.1147903Z 2025-10-10T01:44:30.1148848Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:30.1149456Z 2025-10-10T01:44:30.1149841Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:30.1150233Z 2025-10-10T01:44:30.1150587Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1150935Z 2025-10-10T01:44:30.1151538Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:30.1152228Z 2025-10-10T01:44:30.1152791Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:30.1153216Z 2025-10-10T01:44:30.1153852Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:30.1154440Z 2025-10-10T01:44:30.1154904Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:30.1155343Z 2025-10-10T01:44:30.1155640Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1156006Z 2025-10-10T01:44:30.1156990Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:30.1157853Z 2025-10-10T01:44:30.1158261Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] self._init_executor() 2025-10-10T01:44:30.1158767Z 2025-10-10T01:44:30.1159881Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:30.1160532Z 2025-10-10T01:44:30.1161071Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:30.1161435Z 2025-10-10T01:44:30.1162190Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:30.1162710Z 2025-10-10T01:44:30.1163065Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:30.1163405Z 2025-10-10T01:44:30.1163693Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1163981Z 2025-10-10T01:44:30.1164471Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:30.1164932Z 2025-10-10T01:44:30.1165203Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:44:30.1165480Z 2025-10-10T01:44:30.1165741Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1165997Z 2025-10-10T01:44:30.1166526Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:30.1167008Z 2025-10-10T01:44:30.1167297Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:30.1167653Z 2025-10-10T01:44:30.1167921Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1168195Z 2025-10-10T01:44:30.1168718Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:30.1169211Z 2025-10-10T01:44:30.1169512Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:44:30.1169813Z 2025-10-10T01:44:30.1170087Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1170367Z 2025-10-10T01:44:30.1170811Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:30.1171220Z 2025-10-10T01:44:30.1171572Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:30.1171907Z 2025-10-10T01:44:30.1172189Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1172548Z 2025-10-10T01:44:30.1172891Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:44:30.1173220Z 2025-10-10T01:44:30.1173562Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:44:30.1173903Z 2025-10-10T01:44:30.1174284Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:30.1174649Z 2025-10-10T01:44:30.1174985Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:44:30.1175314Z 2025-10-10T01:44:30.1175676Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:44:30.1176031Z 2025-10-10T01:44:30.1176492Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:30.1176857Z 2025-10-10T01:44:30.1177362Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:30.1177818Z 2025-10-10T01:44:30.1178175Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:30.1178522Z 2025-10-10T01:44:30.1179035Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:30.1179520Z 2025-10-10T01:44:30.1179892Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:30.1180253Z 2025-10-10T01:44:30.1180800Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:30.1181307Z 2025-10-10T01:44:30.1181594Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:44:30.1181886Z 2025-10-10T01:44:30.1182514Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:30.1183124Z 2025-10-10T01:44:30.1183497Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:30.1183847Z 2025-10-10T01:44:30.1184109Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1184381Z 2025-10-10T01:44:30.1184948Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:30.1185460Z 2025-10-10T01:44:30.1185751Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:44:30.1186051Z 2025-10-10T01:44:30.1186324Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1186599Z 2025-10-10T01:44:30.1187190Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:30.1187791Z 2025-10-10T01:44:30.1188072Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:44:30.1188362Z 2025-10-10T01:44:30.1188617Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1188880Z 2025-10-10T01:44:30.1189460Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:30.1189984Z 2025-10-10T01:44:30.1190303Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:30.1190613Z 2025-10-10T01:44:30.1190893Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1191175Z 2025-10-10T01:44:30.1191767Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:30.1192257Z 2025-10-10T01:44:30.1192538Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:44:30.1192829Z 2025-10-10T01:44:30.1193088Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1193364Z 2025-10-10T01:44:30.1193881Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:30.1194361Z 2025-10-10T01:44:30.1194664Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:30.1194967Z 2025-10-10T01:44:30.1195201Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:44:30.1195439Z 2025-10-10T01:44:30.1195923Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:30.1196543Z 2025-10-10T01:44:30.1196803Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] raise RuntimeError( 2025-10-10T01:44:30.1197161Z 2025-10-10T01:44:30.1197731Z (EngineCore_DP0 pid=4568) ERROR 10-10 01:44:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:30.1198442Z (EngineCore_DP0 pid=4568) Process EngineCore_DP0: 2025-10-10T01:44:30.1198846Z (EngineCore_DP0 pid=4568) Traceback (most recent call last): 2025-10-10T01:44:30.1199602Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:44:30.1200134Z (EngineCore_DP0 pid=4568) self.run() 2025-10-10T01:44:30.1200663Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:44:30.1201229Z (EngineCore_DP0 pid=4568) self._target(*self._args, **self._kwargs) 2025-10-10T01:44:30.1201899Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:44:30.1202451Z (EngineCore_DP0 pid=4568) raise e 2025-10-10T01:44:30.1203033Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:30.1203758Z (EngineCore_DP0 pid=4568) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:30.1204200Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1204825Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:30.1205460Z (EngineCore_DP0 pid=4568) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:30.1206114Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:30.1206723Z (EngineCore_DP0 pid=4568) self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:30.1207175Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1207949Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:30.1208546Z (EngineCore_DP0 pid=4568) self._init_executor() 2025-10-10T01:44:30.1209216Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:30.1209891Z (EngineCore_DP0 pid=4568) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:30.1210610Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:30.1211333Z (EngineCore_DP0 pid=4568) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:30.1211830Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1212474Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:30.1213053Z (EngineCore_DP0 pid=4568) return func(*args, **kwargs) 2025-10-10T01:44:30.1213428Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1214058Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:30.1214684Z (EngineCore_DP0 pid=4568) worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:30.1215170Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1215835Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:30.1216498Z (EngineCore_DP0 pid=4568) module = importlib.import_module(module_name) 2025-10-10T01:44:30.1216946Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1217524Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:30.1218147Z (EngineCore_DP0 pid=4568) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:30.1218638Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1219125Z (EngineCore_DP0 pid=4568) File "", line 1387, in _gcd_import 2025-10-10T01:44:30.1219690Z (EngineCore_DP0 pid=4568) File "", line 1360, in _find_and_load 2025-10-10T01:44:30.1220277Z (EngineCore_DP0 pid=4568) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:30.1220908Z (EngineCore_DP0 pid=4568) File "", line 935, in _load_unlocked 2025-10-10T01:44:30.1221477Z (EngineCore_DP0 pid=4568) File "", line 999, in exec_module 2025-10-10T01:44:30.1222070Z (EngineCore_DP0 pid=4568) File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:30.1222810Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:30.1223487Z (EngineCore_DP0 pid=4568) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:30.1224206Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:30.1224941Z (EngineCore_DP0 pid=4568) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:30.1225827Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:30.1226517Z (EngineCore_DP0 pid=4568) class FlashAttentionMetadataBuilder( 2025-10-10T01:44:30.1227309Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:30.1228124Z (EngineCore_DP0 pid=4568) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:30.1228606Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1229294Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:30.1229970Z (EngineCore_DP0 pid=4568) if not is_fa_version_supported(fa_version): 2025-10-10T01:44:30.1230416Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1231161Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:30.1231872Z (EngineCore_DP0 pid=4568) return _is_fa2_supported(device)[0] 2025-10-10T01:44:30.1232287Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1233056Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:30.1233770Z (EngineCore_DP0 pid=4568) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:30.1234221Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1234890Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:30.1235519Z (EngineCore_DP0 pid=4568) prop = get_device_properties(device) 2025-10-10T01:44:30.1235924Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:30.1236577Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:30.1237234Z (EngineCore_DP0 pid=4568) _lazy_init() # will define _get_device_properties 2025-10-10T01:44:30.1237625Z (EngineCore_DP0 pid=4568) ^^^^^^^^^^^^ 2025-10-10T01:44:30.1238202Z (EngineCore_DP0 pid=4568) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:30.1238802Z (EngineCore_DP0 pid=4568) raise RuntimeError( 2025-10-10T01:44:30.1239558Z (EngineCore_DP0 pid=4568) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:30.5196039Z FAILED 2025-10-10T01:44:30.5325483Z models/test_initialization.py::test_can_initialize_large_subset[InternS1ForConditionalGeneration] Fork a new process to run a test 4572 2025-10-10T01:44:30.5335747Z Fork a new process to run a test 0 2025-10-10T01:44:30.5620948Z INFO 10-10 01:44:30 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternS1ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'internlm/Intern-S1'} 2025-10-10T01:44:30.7417497Z 2025-10-10T01:44:30.7419559Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:30.7420223Z config.json: 2.62kB [00:00, 13.7MB/s] 2025-10-10T01:44:30.8169257Z 2025-10-10T01:44:30.8170908Z configuration_interns1.py: 0.00B [00:00, ?B/s] 2025-10-10T01:44:30.8171417Z configuration_interns1.py: 10.2kB [00:00, 71.9MB/s] 2025-10-10T01:44:30.8263655Z A new version of the following files was downloaded from https://huggingface.co/internlm/Intern-S1: 2025-10-10T01:44:30.8264191Z - configuration_interns1.py 2025-10-10T01:44:30.8264804Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:44:31.0547554Z 2025-10-10T01:44:31.0548347Z preprocessor_config.json: 0% 0.00/694 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:44:39.4855340Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] EngineCore failed to start. 2025-10-10T01:44:39.4855685Z 2025-10-10T01:44:39.4856000Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] Traceback (most recent call last): 2025-10-10T01:44:39.4856287Z 2025-10-10T01:44:39.4856825Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:39.4857334Z 2025-10-10T01:44:39.4857640Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:39.4858253Z 2025-10-10T01:44:39.4858567Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4858860Z 2025-10-10T01:44:39.4859379Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:39.4859831Z 2025-10-10T01:44:39.4860157Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:39.4860628Z 2025-10-10T01:44:39.4861116Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:39.4861586Z 2025-10-10T01:44:39.4861937Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:39.4862251Z 2025-10-10T01:44:39.4862725Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4863029Z 2025-10-10T01:44:39.4863566Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:39.4864265Z 2025-10-10T01:44:39.4864712Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] self._init_executor() 2025-10-10T01:44:39.4865201Z 2025-10-10T01:44:39.4866098Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:39.4867030Z 2025-10-10T01:44:39.4867414Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:39.4867752Z 2025-10-10T01:44:39.4868305Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:39.4868809Z 2025-10-10T01:44:39.4869155Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:39.4869484Z 2025-10-10T01:44:39.4869774Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4870191Z 2025-10-10T01:44:39.4870692Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:39.4871141Z 2025-10-10T01:44:39.4871417Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:44:39.4871691Z 2025-10-10T01:44:39.4871944Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4872209Z 2025-10-10T01:44:39.4872716Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:39.4873190Z 2025-10-10T01:44:39.4873481Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:39.4873769Z 2025-10-10T01:44:39.4874039Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4874312Z 2025-10-10T01:44:39.4874841Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:39.4875394Z 2025-10-10T01:44:39.4875719Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:44:39.4876021Z 2025-10-10T01:44:39.4876312Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4876587Z 2025-10-10T01:44:39.4877035Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:39.4877444Z 2025-10-10T01:44:39.4877786Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:39.4878119Z 2025-10-10T01:44:39.4878402Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4878687Z 2025-10-10T01:44:39.4879259Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:44:39.4879620Z 2025-10-10T01:44:39.4879982Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:44:39.4880360Z 2025-10-10T01:44:39.4880739Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:39.4881101Z 2025-10-10T01:44:39.4881446Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:44:39.4881784Z 2025-10-10T01:44:39.4882143Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:44:39.4882499Z 2025-10-10T01:44:39.4882878Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:39.4883251Z 2025-10-10T01:44:39.4883754Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:39.4884215Z 2025-10-10T01:44:39.4884564Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:39.4884954Z 2025-10-10T01:44:39.4885478Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:39.4885955Z 2025-10-10T01:44:39.4886330Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:39.4886692Z 2025-10-10T01:44:39.4887251Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:39.4887761Z 2025-10-10T01:44:39.4888050Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:44:39.4888346Z 2025-10-10T01:44:39.4888967Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:39.4889539Z 2025-10-10T01:44:39.4889903Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:39.4890303Z 2025-10-10T01:44:39.4890561Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4890827Z 2025-10-10T01:44:39.4891381Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:39.4891891Z 2025-10-10T01:44:39.4892189Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:44:39.4892480Z 2025-10-10T01:44:39.4892754Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4893033Z 2025-10-10T01:44:39.4893642Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:39.4894192Z 2025-10-10T01:44:39.4901194Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:44:39.4901573Z 2025-10-10T01:44:39.4901926Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4902215Z 2025-10-10T01:44:39.4902832Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:39.4903384Z 2025-10-10T01:44:39.4903716Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:39.4904058Z 2025-10-10T01:44:39.4904345Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4904636Z 2025-10-10T01:44:39.4905180Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:39.4905675Z 2025-10-10T01:44:39.4905969Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:44:39.4906261Z 2025-10-10T01:44:39.4906531Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4906797Z 2025-10-10T01:44:39.4907346Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:39.4907919Z 2025-10-10T01:44:39.4908246Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:39.4908566Z 2025-10-10T01:44:39.4908801Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:44:39.4909050Z 2025-10-10T01:44:39.4909534Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:39.4909998Z 2025-10-10T01:44:39.4910248Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] raise RuntimeError( 2025-10-10T01:44:39.4910509Z 2025-10-10T01:44:39.4911075Z (EngineCore_DP0 pid=4651) ERROR 10-10 01:44:39 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:39.4911774Z (EngineCore_DP0 pid=4651) Process EngineCore_DP0: 2025-10-10T01:44:39.4912167Z (EngineCore_DP0 pid=4651) Traceback (most recent call last): 2025-10-10T01:44:39.4912863Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:44:39.4913394Z (EngineCore_DP0 pid=4651) self.run() 2025-10-10T01:44:39.4913928Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:44:39.4914494Z (EngineCore_DP0 pid=4651) self._target(*self._args, **self._kwargs) 2025-10-10T01:44:39.4915165Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:44:39.4915724Z (EngineCore_DP0 pid=4651) raise e 2025-10-10T01:44:39.4916313Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:39.4916966Z (EngineCore_DP0 pid=4651) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:39.4917520Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4918171Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:39.4918815Z (EngineCore_DP0 pid=4651) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:39.4919561Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:39.4920185Z (EngineCore_DP0 pid=4651) self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:39.4920640Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4921294Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:39.4921881Z (EngineCore_DP0 pid=4651) self._init_executor() 2025-10-10T01:44:39.4922548Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:39.4923246Z (EngineCore_DP0 pid=4651) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:39.4923967Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:39.4924749Z (EngineCore_DP0 pid=4651) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:39.4925258Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4925899Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:39.4926499Z (EngineCore_DP0 pid=4651) return func(*args, **kwargs) 2025-10-10T01:44:39.4926887Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4927767Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:39.4928445Z (EngineCore_DP0 pid=4651) worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:39.4928868Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4929546Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:39.4930218Z (EngineCore_DP0 pid=4651) module = importlib.import_module(module_name) 2025-10-10T01:44:39.4930752Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4931348Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:39.4931972Z (EngineCore_DP0 pid=4651) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:39.4932470Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4932971Z (EngineCore_DP0 pid=4651) File "", line 1387, in _gcd_import 2025-10-10T01:44:39.4933532Z (EngineCore_DP0 pid=4651) File "", line 1360, in _find_and_load 2025-10-10T01:44:39.4934124Z (EngineCore_DP0 pid=4651) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:39.4934699Z (EngineCore_DP0 pid=4651) File "", line 935, in _load_unlocked 2025-10-10T01:44:39.4935378Z (EngineCore_DP0 pid=4651) File "", line 999, in exec_module 2025-10-10T01:44:39.4936006Z (EngineCore_DP0 pid=4651) File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:39.4936740Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:39.4937424Z (EngineCore_DP0 pid=4651) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:39.4938149Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:39.4938876Z (EngineCore_DP0 pid=4651) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:39.4939675Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:39.4940376Z (EngineCore_DP0 pid=4651) class FlashAttentionMetadataBuilder( 2025-10-10T01:44:39.4941172Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:39.4941983Z (EngineCore_DP0 pid=4651) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:39.4942458Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4943222Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:39.4943903Z (EngineCore_DP0 pid=4651) if not is_fa_version_supported(fa_version): 2025-10-10T01:44:39.4944347Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4945097Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:39.4945793Z (EngineCore_DP0 pid=4651) return _is_fa2_supported(device)[0] 2025-10-10T01:44:39.4946208Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4946917Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:39.4947634Z (EngineCore_DP0 pid=4651) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:39.4948085Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4948797Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:39.4949451Z (EngineCore_DP0 pid=4651) prop = get_device_properties(device) 2025-10-10T01:44:39.4949870Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:39.4950526Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:39.4951200Z (EngineCore_DP0 pid=4651) _lazy_init() # will define _get_device_properties 2025-10-10T01:44:39.4951610Z (EngineCore_DP0 pid=4651) ^^^^^^^^^^^^ 2025-10-10T01:44:39.4952211Z (EngineCore_DP0 pid=4651) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:39.4952783Z (EngineCore_DP0 pid=4651) raise RuntimeError( 2025-10-10T01:44:39.4953595Z (EngineCore_DP0 pid=4651) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:39.8964470Z FAILED 2025-10-10T01:44:39.9093506Z models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEForSequenceClassification] Fork a new process to run a test 4655 2025-10-10T01:44:39.9104445Z Fork a new process to run a test 0 2025-10-10T01:44:39.9108577Z `transformers==4.56.2` installed, but `transformers>=4.57.0.dev0` is required to run this model. 2025-10-10T01:44:40.2090289Z PASSED 2025-10-10T01:44:40.2219998Z models/test_initialization.py::test_can_initialize_large_subset[PaliGemmaForConditionalGeneration] Fork a new process to run a test 4656 2025-10-10T01:44:40.2231504Z Fork a new process to run a test 0 2025-10-10T01:44:40.2503269Z INFO 10-10 01:44:40 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='PaliGemmaForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/paligemma-3b-mix-224'} 2025-10-10T01:44:40.3634736Z 2025-10-10T01:44:40.3636627Z config.json: 0% 0.00/1.03k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:44:50.1278509Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] EngineCore failed to start. 2025-10-10T01:44:50.1278943Z 2025-10-10T01:44:50.1279575Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] Traceback (most recent call last): 2025-10-10T01:44:50.1280027Z 2025-10-10T01:44:50.1280720Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:50.1281311Z 2025-10-10T01:44:50.1281700Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:50.1282075Z 2025-10-10T01:44:50.1282428Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1282780Z 2025-10-10T01:44:50.1283390Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:50.1284102Z 2025-10-10T01:44:50.1284456Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:50.1284784Z 2025-10-10T01:44:50.1285389Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:50.1285911Z 2025-10-10T01:44:50.1286240Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:50.1286555Z 2025-10-10T01:44:50.1286835Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1287134Z 2025-10-10T01:44:50.1287645Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:50.1288115Z 2025-10-10T01:44:50.1288559Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] self._init_executor() 2025-10-10T01:44:50.1288847Z 2025-10-10T01:44:50.1289656Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:50.1290445Z 2025-10-10T01:44:50.1291066Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:50.1291609Z 2025-10-10T01:44:50.1292666Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:50.1293468Z 2025-10-10T01:44:50.1294076Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:50.1294434Z 2025-10-10T01:44:50.1294742Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1295029Z 2025-10-10T01:44:50.1295521Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:50.1295972Z 2025-10-10T01:44:50.1296475Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:44:50.1296943Z 2025-10-10T01:44:50.1297214Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1297480Z 2025-10-10T01:44:50.1298011Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:50.1298506Z 2025-10-10T01:44:50.1298799Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:50.1299093Z 2025-10-10T01:44:50.1299357Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1299623Z 2025-10-10T01:44:50.1300157Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:50.1300640Z 2025-10-10T01:44:50.1300948Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:44:50.1301246Z 2025-10-10T01:44:50.1301523Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1301869Z 2025-10-10T01:44:50.1302323Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:50.1302732Z 2025-10-10T01:44:50.1303076Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:50.1303408Z 2025-10-10T01:44:50.1303688Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1303968Z 2025-10-10T01:44:50.1304322Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:44:50.1304659Z 2025-10-10T01:44:50.1305004Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:44:50.1305341Z 2025-10-10T01:44:50.1305848Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:50.1306205Z 2025-10-10T01:44:50.1306576Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:44:50.1306906Z 2025-10-10T01:44:50.1307270Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:44:50.1307616Z 2025-10-10T01:44:50.1307994Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:50.1308348Z 2025-10-10T01:44:50.1308842Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:50.1309307Z 2025-10-10T01:44:50.1309659Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:50.1309992Z 2025-10-10T01:44:50.1310504Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:50.1310980Z 2025-10-10T01:44:50.1311359Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:50.1311771Z 2025-10-10T01:44:50.1312329Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:50.1312830Z 2025-10-10T01:44:50.1313137Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:44:50.1313432Z 2025-10-10T01:44:50.1314058Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:50.1314623Z 2025-10-10T01:44:50.1314990Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:50.1315338Z 2025-10-10T01:44:50.1315588Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1315855Z 2025-10-10T01:44:50.1316411Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:50.1316983Z 2025-10-10T01:44:50.1317286Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:44:50.1317582Z 2025-10-10T01:44:50.1317851Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1318123Z 2025-10-10T01:44:50.1318719Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:50.1319393Z 2025-10-10T01:44:50.1319688Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:44:50.1319969Z 2025-10-10T01:44:50.1320232Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1320495Z 2025-10-10T01:44:50.1321209Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:50.1321736Z 2025-10-10T01:44:50.1322061Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:50.1322376Z 2025-10-10T01:44:50.1322658Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1322937Z 2025-10-10T01:44:50.1323468Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:50.1323957Z 2025-10-10T01:44:50.1324234Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:44:50.1324529Z 2025-10-10T01:44:50.1324794Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1325060Z 2025-10-10T01:44:50.1325581Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:50.1326058Z 2025-10-10T01:44:50.1326387Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:50.1326689Z 2025-10-10T01:44:50.1326983Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:44:50.1327224Z 2025-10-10T01:44:50.1327707Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:50.1328161Z 2025-10-10T01:44:50.1328412Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] raise RuntimeError( 2025-10-10T01:44:50.1328671Z 2025-10-10T01:44:50.1329241Z (EngineCore_DP0 pid=4736) ERROR 10-10 01:44:50 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:50.1329935Z (EngineCore_DP0 pid=4736) Process EngineCore_DP0: 2025-10-10T01:44:50.1330328Z (EngineCore_DP0 pid=4736) Traceback (most recent call last): 2025-10-10T01:44:50.1330931Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:44:50.1331445Z (EngineCore_DP0 pid=4736) self.run() 2025-10-10T01:44:50.1332010Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:44:50.1332631Z (EngineCore_DP0 pid=4736) self._target(*self._args, **self._kwargs) 2025-10-10T01:44:50.1333309Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:44:50.1333857Z (EngineCore_DP0 pid=4736) raise e 2025-10-10T01:44:50.1334436Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:44:50.1335066Z (EngineCore_DP0 pid=4736) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:44:50.1335507Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1336136Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:44:50.1336775Z (EngineCore_DP0 pid=4736) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:44:50.1337510Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:44:50.1338135Z (EngineCore_DP0 pid=4736) self.model_executor = executor_class(vllm_config) 2025-10-10T01:44:50.1338587Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1339245Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:44:50.1339840Z (EngineCore_DP0 pid=4736) self._init_executor() 2025-10-10T01:44:50.1340501Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:44:50.1341181Z (EngineCore_DP0 pid=4736) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:44:50.1341911Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:44:50.1342621Z (EngineCore_DP0 pid=4736) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:44:50.1343119Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1343760Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:44:50.1344401Z (EngineCore_DP0 pid=4736) return func(*args, **kwargs) 2025-10-10T01:44:50.1344784Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1345419Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:44:50.1346054Z (EngineCore_DP0 pid=4736) worker_class = resolve_obj_by_qualname( 2025-10-10T01:44:50.1346480Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1347140Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:44:50.1347799Z (EngineCore_DP0 pid=4736) module = importlib.import_module(module_name) 2025-10-10T01:44:50.1348243Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1348829Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:44:50.1349452Z (EngineCore_DP0 pid=4736) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:44:50.1350006Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1350512Z (EngineCore_DP0 pid=4736) File "", line 1387, in _gcd_import 2025-10-10T01:44:50.1351063Z (EngineCore_DP0 pid=4736) File "", line 1360, in _find_and_load 2025-10-10T01:44:50.1351652Z (EngineCore_DP0 pid=4736) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:50.1352225Z (EngineCore_DP0 pid=4736) File "", line 935, in _load_unlocked 2025-10-10T01:44:50.1352791Z (EngineCore_DP0 pid=4736) File "", line 999, in exec_module 2025-10-10T01:44:50.1353407Z (EngineCore_DP0 pid=4736) File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:50.1354138Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:44:50.1354903Z (EngineCore_DP0 pid=4736) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:44:50.1355628Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:44:50.1356367Z (EngineCore_DP0 pid=4736) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:44:50.1357142Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:44:50.1357841Z (EngineCore_DP0 pid=4736) class FlashAttentionMetadataBuilder( 2025-10-10T01:44:50.1358659Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:44:50.1359547Z (EngineCore_DP0 pid=4736) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:44:50.1360033Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1360717Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:44:50.1361422Z (EngineCore_DP0 pid=4736) if not is_fa_version_supported(fa_version): 2025-10-10T01:44:50.1361859Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1362651Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:44:50.1363363Z (EngineCore_DP0 pid=4736) return _is_fa2_supported(device)[0] 2025-10-10T01:44:50.1363784Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1364493Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:44:50.1365202Z (EngineCore_DP0 pid=4736) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:44:50.1365652Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1366318Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:50.1366960Z (EngineCore_DP0 pid=4736) prop = get_device_properties(device) 2025-10-10T01:44:50.1367371Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:50.1368096Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:50.1368760Z (EngineCore_DP0 pid=4736) _lazy_init() # will define _get_device_properties 2025-10-10T01:44:50.1369152Z (EngineCore_DP0 pid=4736) ^^^^^^^^^^^^ 2025-10-10T01:44:50.1369731Z (EngineCore_DP0 pid=4736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:50.1370290Z (EngineCore_DP0 pid=4736) raise RuntimeError( 2025-10-10T01:44:50.1370989Z (EngineCore_DP0 pid=4736) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:50.5491212Z FAILED 2025-10-10T01:44:50.5620162Z models/test_initialization.py::test_can_initialize_large_subset[RForConditionalGeneration] Fork a new process to run a test 4740 2025-10-10T01:44:50.5630898Z Fork a new process to run a test 0 2025-10-10T01:44:50.5911921Z INFO 10-10 01:44:50 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='RForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'YannQi/R-4B'} 2025-10-10T01:44:50.8144435Z 2025-10-10T01:44:50.8146380Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:44:50.8146688Z config.json: 2.19kB [00:00, 11.7MB/s] 2025-10-10T01:44:50.8827395Z 2025-10-10T01:44:50.8828370Z configuration_r.py: 0.00B [00:00, ?B/s] 2025-10-10T01:44:50.8828735Z configuration_r.py: 3.59kB [00:00, 36.6MB/s] 2025-10-10T01:44:50.8919331Z A new version of the following files was downloaded from https://huggingface.co/YannQi/R-4B: 2025-10-10T01:44:50.8919814Z - configuration_r.py 2025-10-10T01:44:50.8920439Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:44:51.1258032Z 2025-10-10T01:44:51.1258789Z preprocessor_config.json: 0% 0.00/745 [00:00", line 1387, in _gcd_import 2025-10-10T01:44:59.5638941Z 2025-10-10T01:44:59.5639327Z ERROR 10-10 01:44:59 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:44:59.5639686Z 2025-10-10T01:44:59.5640086Z ERROR 10-10 01:44:59 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:44:59.5640466Z 2025-10-10T01:44:59.5640723Z ERROR 10-10 01:44:59 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:44:59.5641066Z 2025-10-10T01:44:59.5641338Z ERROR 10-10 01:44:59 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:44:59.5641658Z 2025-10-10T01:44:59.5641887Z ERROR 10-10 01:44:59 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:44:59.5642194Z 2025-10-10T01:44:59.5642547Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/rvl.py", line 14, in 2025-10-10T01:44:59.5642968Z 2025-10-10T01:44:59.5643230Z ERROR 10-10 01:44:59 [registry.py:542] from .llava_next import (LlavaDummyInputsBuilder, LlavaNextMultiModalProcessor, 2025-10-10T01:44:59.5643568Z 2025-10-10T01:44:59.5644040Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava_next.py", line 24, in 2025-10-10T01:44:59.5644666Z 2025-10-10T01:44:59.5645031Z ERROR 10-10 01:44:59 [registry.py:542] from .llava import (BaseLlavaMultiModalProcessor, BaseLlavaProcessingInfo, 2025-10-10T01:44:59.5645633Z 2025-10-10T01:44:59.5645997Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava.py", line 40, in 2025-10-10T01:44:59.5646434Z 2025-10-10T01:44:59.5646655Z ERROR 10-10 01:44:59 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T01:44:59.5646938Z 2025-10-10T01:44:59.5647296Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:44:59.5647725Z 2025-10-10T01:44:59.5647866Z ERROR 10-10 01:44:59 [registry.py:542] from xformers import ops as xops 2025-10-10T01:44:59.5648101Z 2025-10-10T01:44:59.5648412Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:44:59.5648801Z 2025-10-10T01:44:59.5648919Z ERROR 10-10 01:44:59 [registry.py:542] from .fmha import ( 2025-10-10T01:44:59.5649113Z 2025-10-10T01:44:59.5649530Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:44:59.5649943Z 2025-10-10T01:44:59.5650180Z ERROR 10-10 01:44:59 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:44:59.5650477Z 2025-10-10T01:44:59.5650836Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:44:59.5651259Z 2025-10-10T01:44:59.5651416Z ERROR 10-10 01:44:59 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:44:59.5651643Z 2025-10-10T01:44:59.5651783Z ERROR 10-10 01:44:59 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:59.5651999Z 2025-10-10T01:44:59.5652313Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:44:59.5652705Z 2025-10-10T01:44:59.5652811Z ERROR 10-10 01:44:59 [registry.py:542] value = func() 2025-10-10T01:44:59.5652989Z 2025-10-10T01:44:59.5653091Z ERROR 10-10 01:44:59 [registry.py:542] ^^^^^^ 2025-10-10T01:44:59.5653268Z 2025-10-10T01:44:59.5653610Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:44:59.5654018Z 2025-10-10T01:44:59.5654244Z ERROR 10-10 01:44:59 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:44:59.5654496Z 2025-10-10T01:44:59.5654630Z ERROR 10-10 01:44:59 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:59.5654844Z 2025-10-10T01:44:59.5655191Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:44:59.5655622Z 2025-10-10T01:44:59.5655770Z ERROR 10-10 01:44:59 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:44:59.5655998Z 2025-10-10T01:44:59.5656126Z ERROR 10-10 01:44:59 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:44:59.5656324Z 2025-10-10T01:44:59.5656675Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:44:59.5657092Z 2025-10-10T01:44:59.5657271Z ERROR 10-10 01:44:59 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:44:59.5657515Z 2025-10-10T01:44:59.5657617Z ERROR 10-10 01:44:59 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:44:59.5657799Z 2025-10-10T01:44:59.5658125Z ERROR 10-10 01:44:59 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:44:59.5658569Z 2025-10-10T01:44:59.5658695Z ERROR 10-10 01:44:59 [registry.py:542] raise RuntimeError( 2025-10-10T01:44:59.5658881Z 2025-10-10T01:44:59.5659274Z ERROR 10-10 01:44:59 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:44:59.9494911Z FAILED 2025-10-10T01:44:59.9624078Z models/test_initialization.py::test_can_initialize_large_subset[DeciLMForCausalLM] Fork a new process to run a test 4817 2025-10-10T01:44:59.9634843Z Fork a new process to run a test 0 2025-10-10T01:44:59.9913692Z INFO 10-10 01:44:59 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DeciLMForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nvidia/Llama-3_3-Nemotron-Super-49B-v1'} 2025-10-10T01:45:00.1446566Z 2025-10-10T01:45:00.1451128Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:45:00.1451566Z config.json: 37.1kB [00:00, 93.9MB/s] 2025-10-10T01:45:00.2282582Z 2025-10-10T01:45:00.2284503Z configuration_decilm.py: 0.00B [00:00, ?B/s] 2025-10-10T01:45:00.2284892Z configuration_decilm.py: 2.57kB [00:00, 10.7MB/s] 2025-10-10T01:45:00.2737575Z 2025-10-10T01:45:00.2746050Z (…)nsformers_4_44_2__configuration_llama.py: 0.00B [00:00, ?B/s] 2025-10-10T01:45:00.2746612Z (…)nsformers_4_44_2__configuration_llama.py: 11.1kB [00:00, 14.3MB/s] 2025-10-10T01:45:00.3135030Z 2025-10-10T01:45:00.3138307Z (…)nsformers_4_44_2__modeling_rope_utils.py: 0.00B [00:00, ?B/s] 2025-10-10T01:45:00.3138831Z (…)nsformers_4_44_2__modeling_rope_utils.py: 28.1kB [00:00, 110MB/s] 2025-10-10T01:45:00.3219017Z A new version of the following files was downloaded from https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1: 2025-10-10T01:45:00.3219798Z - transformers_4_44_2__modeling_rope_utils.py 2025-10-10T01:45:00.3220621Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:45:00.3221675Z A new version of the following files was downloaded from https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1: 2025-10-10T01:45:00.3222384Z - transformers_4_44_2__configuration_llama.py 2025-10-10T01:45:00.3222659Z - transformers_4_44_2__modeling_rope_utils.py 2025-10-10T01:45:00.3223278Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:45:00.3695310Z 2025-10-10T01:45:00.3697397Z block_config.py: 0.00B [00:00, ?B/s] 2025-10-10T01:45:00.3723600Z block_config.py: 4.35kB [00:00, 30.4MB/s] 2025-10-10T01:45:00.3724210Z A new version of the following files was downloaded from https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1: 2025-10-10T01:45:00.3724789Z - block_config.py 2025-10-10T01:45:00.3725375Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:45:00.3726277Z A new version of the following files was downloaded from https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1: 2025-10-10T01:45:00.3726821Z - configuration_decilm.py 2025-10-10T01:45:00.3727097Z - transformers_4_44_2__configuration_llama.py 2025-10-10T01:45:00.3727381Z - block_config.py 2025-10-10T01:45:00.3727923Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:45:07.2661676Z INFO 10-10 01:45:07 [model.py:551] Resolved architecture: DeciLMForCausalLM 2025-10-10T01:45:07.2662153Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:45:07.2908802Z INFO 10-10 01:45:07 [model.py:1545] Using max model len 131072 2025-10-10T01:45:07.2910460Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:45:07.3589633Z INFO 10-10 01:45:07 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:45:07.4012085Z 2025-10-10T01:45:07.4014923Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:45:07.4015265Z tokenizer_config.json: 51.3kB [00:00, 222MB/s] 2025-10-10T01:45:07.5202003Z 2025-10-10T01:45:07.7463895Z tokenizer.json: 0% 0.00/17.2M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:08.6150436Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:08.6150943Z 2025-10-10T01:45:08.6151452Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:08.6151846Z 2025-10-10T01:45:08.6152682Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:08.6153533Z 2025-10-10T01:45:08.6153961Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:08.6154340Z 2025-10-10T01:45:08.6154691Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6155040Z 2025-10-10T01:45:08.6155637Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:08.6156218Z 2025-10-10T01:45:08.6156759Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:08.6157205Z 2025-10-10T01:45:08.6157962Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:08.6158531Z 2025-10-10T01:45:08.6159247Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:08.6159653Z 2025-10-10T01:45:08.6160032Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6160412Z 2025-10-10T01:45:08.6160989Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:08.6161581Z 2025-10-10T01:45:08.6161867Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] self._init_executor() 2025-10-10T01:45:08.6162133Z 2025-10-10T01:45:08.6162784Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:08.6163501Z 2025-10-10T01:45:08.6164007Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:08.6164347Z 2025-10-10T01:45:08.6165005Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:08.6165513Z 2025-10-10T01:45:08.6165863Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:08.6166318Z 2025-10-10T01:45:08.6166608Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6166885Z 2025-10-10T01:45:08.6167372Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:08.6167823Z 2025-10-10T01:45:08.6168091Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:08.6168371Z 2025-10-10T01:45:08.6168632Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6168889Z 2025-10-10T01:45:08.6169397Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:08.6169875Z 2025-10-10T01:45:08.6170170Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:08.6170468Z 2025-10-10T01:45:08.6170734Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6171092Z 2025-10-10T01:45:08.6171628Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:08.6172111Z 2025-10-10T01:45:08.6172411Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:08.6172708Z 2025-10-10T01:45:08.6172981Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6173251Z 2025-10-10T01:45:08.6173692Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:08.6174096Z 2025-10-10T01:45:08.6174436Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:08.6174763Z 2025-10-10T01:45:08.6175119Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6175404Z 2025-10-10T01:45:08.6175744Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:08.6176074Z 2025-10-10T01:45:08.6176416Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:08.6176753Z 2025-10-10T01:45:08.6177123Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:08.6177484Z 2025-10-10T01:45:08.6177821Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:08.6178147Z 2025-10-10T01:45:08.6178512Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:08.6178859Z 2025-10-10T01:45:08.6179231Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:08.6179584Z 2025-10-10T01:45:08.6180088Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:08.6180602Z 2025-10-10T01:45:08.6180956Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:08.6181300Z 2025-10-10T01:45:08.6181823Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:08.6182307Z 2025-10-10T01:45:08.6182685Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:08.6183050Z 2025-10-10T01:45:08.6183603Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:08.6184129Z 2025-10-10T01:45:08.6184419Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:08.6184721Z 2025-10-10T01:45:08.6185345Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:08.6185957Z 2025-10-10T01:45:08.6186332Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:08.6186679Z 2025-10-10T01:45:08.6186934Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6187191Z 2025-10-10T01:45:08.6187751Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:08.6188262Z 2025-10-10T01:45:08.6188557Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:08.6188853Z 2025-10-10T01:45:08.6189122Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6189403Z 2025-10-10T01:45:08.6190078Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:08.6190637Z 2025-10-10T01:45:08.6190918Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:08.6191204Z 2025-10-10T01:45:08.6191460Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6191724Z 2025-10-10T01:45:08.6192305Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:08.6192830Z 2025-10-10T01:45:08.6193150Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:08.6193460Z 2025-10-10T01:45:08.6193741Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6194012Z 2025-10-10T01:45:08.6194529Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:08.6195016Z 2025-10-10T01:45:08.6195295Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:08.6195585Z 2025-10-10T01:45:08.6195891Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6196431Z 2025-10-10T01:45:08.6196975Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:08.6197471Z 2025-10-10T01:45:08.6197780Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:08.6198083Z 2025-10-10T01:45:08.6198310Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:08.6198546Z 2025-10-10T01:45:08.6199121Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:08.6199571Z 2025-10-10T01:45:08.6199825Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] raise RuntimeError( 2025-10-10T01:45:08.6200088Z 2025-10-10T01:45:08.6200650Z (EngineCore_DP0 pid=4895) ERROR 10-10 01:45:08 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:08.6201464Z (EngineCore_DP0 pid=4895) Process EngineCore_DP0: 2025-10-10T01:45:08.6201857Z (EngineCore_DP0 pid=4895) Traceback (most recent call last): 2025-10-10T01:45:08.6202477Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:08.6203002Z (EngineCore_DP0 pid=4895) self.run() 2025-10-10T01:45:08.6203521Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:08.6204078Z (EngineCore_DP0 pid=4895) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:08.6204754Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:08.6205302Z (EngineCore_DP0 pid=4895) raise e 2025-10-10T01:45:08.6206027Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:08.6206687Z (EngineCore_DP0 pid=4895) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:08.6207131Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6207746Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:08.6208382Z (EngineCore_DP0 pid=4895) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:08.6209045Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:08.6209662Z (EngineCore_DP0 pid=4895) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:08.6210116Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6210767Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:08.6211358Z (EngineCore_DP0 pid=4895) self._init_executor() 2025-10-10T01:45:08.6212017Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:08.6212701Z (EngineCore_DP0 pid=4895) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:08.6213488Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:08.6214189Z (EngineCore_DP0 pid=4895) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:08.6214688Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6215324Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:08.6215908Z (EngineCore_DP0 pid=4895) return func(*args, **kwargs) 2025-10-10T01:45:08.6216294Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6216924Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:08.6217559Z (EngineCore_DP0 pid=4895) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:08.6217999Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6218668Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:08.6219370Z (EngineCore_DP0 pid=4895) module = importlib.import_module(module_name) 2025-10-10T01:45:08.6219822Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6220394Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:08.6221015Z (EngineCore_DP0 pid=4895) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:08.6221511Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6222008Z (EngineCore_DP0 pid=4895) File "", line 1387, in _gcd_import 2025-10-10T01:45:08.6222569Z (EngineCore_DP0 pid=4895) File "", line 1360, in _find_and_load 2025-10-10T01:45:08.6223228Z (EngineCore_DP0 pid=4895) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:08.6223812Z (EngineCore_DP0 pid=4895) File "", line 935, in _load_unlocked 2025-10-10T01:45:08.6224384Z (EngineCore_DP0 pid=4895) File "", line 999, in exec_module 2025-10-10T01:45:08.6224983Z (EngineCore_DP0 pid=4895) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:08.6225723Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:08.6226405Z (EngineCore_DP0 pid=4895) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:08.6227130Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:08.6227851Z (EngineCore_DP0 pid=4895) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:08.6228627Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:08.6229294Z (EngineCore_DP0 pid=4895) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:08.6230084Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:08.6230968Z (EngineCore_DP0 pid=4895) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:08.6231443Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6232137Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:08.6232819Z (EngineCore_DP0 pid=4895) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:08.6233259Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6233994Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:08.6234689Z (EngineCore_DP0 pid=4895) return _is_fa2_supported(device)[0] 2025-10-10T01:45:08.6235101Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6235810Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:08.6236584Z (EngineCore_DP0 pid=4895) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:08.6237051Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6237720Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:08.6238354Z (EngineCore_DP0 pid=4895) prop = get_device_properties(device) 2025-10-10T01:45:08.6238772Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:08.6239506Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:08.6240180Z (EngineCore_DP0 pid=4895) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:08.6240574Z (EngineCore_DP0 pid=4895) ^^^^^^^^^^^^ 2025-10-10T01:45:08.6241229Z (EngineCore_DP0 pid=4895) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:08.6241799Z (EngineCore_DP0 pid=4895) raise RuntimeError( 2025-10-10T01:45:08.6242478Z (EngineCore_DP0 pid=4895) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:09.0349928Z FAILED 2025-10-10T01:45:09.0479899Z models/test_initialization.py::test_can_initialize_large_subset[Qwen3NextForCausalLM] Fork a new process to run a test 4899 2025-10-10T01:45:09.0491069Z Fork a new process to run a test 0 2025-10-10T01:45:09.0494750Z `transformers==4.56.2` installed, but `transformers>=4.56.3` is required to run this model. 2025-10-10T01:45:09.3560388Z PASSED 2025-10-10T01:45:09.3689150Z models/test_initialization.py::test_can_initialize_large_subset[Gemma3ForCausalLM] Fork a new process to run a test 4900 2025-10-10T01:45:09.3700455Z Fork a new process to run a test 0 2025-10-10T01:45:09.3980925Z INFO 10-10 01:45:09 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/gemma-3-1b-it'} 2025-10-10T01:45:09.5128851Z 2025-10-10T01:45:09.5129741Z config.json: 0% 0.00/899 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:19.0487300Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:19.0487724Z 2025-10-10T01:45:19.0488122Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:19.0488503Z 2025-10-10T01:45:19.0489201Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:19.0489796Z 2025-10-10T01:45:19.0490107Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:19.0490419Z 2025-10-10T01:45:19.0490693Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0490982Z 2025-10-10T01:45:19.0491483Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:19.0491940Z 2025-10-10T01:45:19.0492270Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:19.0492848Z 2025-10-10T01:45:19.0493508Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:19.0493950Z 2025-10-10T01:45:19.0494274Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:19.0494576Z 2025-10-10T01:45:19.0494859Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0495133Z 2025-10-10T01:45:19.0495636Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:19.0496325Z 2025-10-10T01:45:19.0496622Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] self._init_executor() 2025-10-10T01:45:19.0496884Z 2025-10-10T01:45:19.0497821Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:19.0498758Z 2025-10-10T01:45:19.0499149Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:19.0499582Z 2025-10-10T01:45:19.0500382Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:19.0501217Z 2025-10-10T01:45:19.0501851Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:19.0502234Z 2025-10-10T01:45:19.0502550Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0502844Z 2025-10-10T01:45:19.0503355Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:19.0503812Z 2025-10-10T01:45:19.0504082Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:19.0504357Z 2025-10-10T01:45:19.0504613Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0504874Z 2025-10-10T01:45:19.0505545Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:19.0506034Z 2025-10-10T01:45:19.0506324Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:19.0506633Z 2025-10-10T01:45:19.0506907Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0507195Z 2025-10-10T01:45:19.0507723Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:19.0508213Z 2025-10-10T01:45:19.0508517Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:19.0508824Z 2025-10-10T01:45:19.0509114Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0509389Z 2025-10-10T01:45:19.0509853Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:19.0510334Z 2025-10-10T01:45:19.0510685Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:19.0511015Z 2025-10-10T01:45:19.0511299Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0511582Z 2025-10-10T01:45:19.0511921Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:19.0512258Z 2025-10-10T01:45:19.0512605Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:19.0512942Z 2025-10-10T01:45:19.0513324Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:19.0513690Z 2025-10-10T01:45:19.0514113Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:19.0514446Z 2025-10-10T01:45:19.0514813Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:19.0515161Z 2025-10-10T01:45:19.0515535Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:19.0515892Z 2025-10-10T01:45:19.0516396Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:19.0516856Z 2025-10-10T01:45:19.0517202Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:19.0517547Z 2025-10-10T01:45:19.0518069Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:19.0518550Z 2025-10-10T01:45:19.0518924Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:19.0519499Z 2025-10-10T01:45:19.0520160Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:19.0520833Z 2025-10-10T01:45:19.0521182Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:19.0521529Z 2025-10-10T01:45:19.0522207Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:19.0522773Z 2025-10-10T01:45:19.0523139Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:19.0523488Z 2025-10-10T01:45:19.0523761Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0524028Z 2025-10-10T01:45:19.0524592Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:19.0525106Z 2025-10-10T01:45:19.0525398Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:19.0525739Z 2025-10-10T01:45:19.0526026Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0526303Z 2025-10-10T01:45:19.0526897Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:19.0527447Z 2025-10-10T01:45:19.0527732Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:19.0528024Z 2025-10-10T01:45:19.0528285Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0528547Z 2025-10-10T01:45:19.0529130Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:19.0529658Z 2025-10-10T01:45:19.0530082Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:19.0530396Z 2025-10-10T01:45:19.0530675Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0530945Z 2025-10-10T01:45:19.0531465Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:19.0531953Z 2025-10-10T01:45:19.0532231Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:19.0532520Z 2025-10-10T01:45:19.0532778Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0533052Z 2025-10-10T01:45:19.0533584Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:19.0534074Z 2025-10-10T01:45:19.0534383Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:19.0534686Z 2025-10-10T01:45:19.0534914Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:19.0535156Z 2025-10-10T01:45:19.0535645Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:19.0536138Z 2025-10-10T01:45:19.0536390Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] raise RuntimeError( 2025-10-10T01:45:19.0536649Z 2025-10-10T01:45:19.0537216Z (EngineCore_DP0 pid=4979) ERROR 10-10 01:45:19 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:19.0537910Z (EngineCore_DP0 pid=4979) Process EngineCore_DP0: 2025-10-10T01:45:19.0538312Z (EngineCore_DP0 pid=4979) Traceback (most recent call last): 2025-10-10T01:45:19.0538937Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:19.0539468Z (EngineCore_DP0 pid=4979) self.run() 2025-10-10T01:45:19.0540004Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:19.0540573Z (EngineCore_DP0 pid=4979) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:19.0541237Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:19.0541843Z (EngineCore_DP0 pid=4979) raise e 2025-10-10T01:45:19.0542434Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:19.0543066Z (EngineCore_DP0 pid=4979) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:19.0543509Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0544126Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:19.0544798Z (EngineCore_DP0 pid=4979) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:19.0545457Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:19.0546152Z (EngineCore_DP0 pid=4979) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:19.0546612Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0547258Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:19.0547842Z (EngineCore_DP0 pid=4979) self._init_executor() 2025-10-10T01:45:19.0548520Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:19.0549270Z (EngineCore_DP0 pid=4979) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:19.0550132Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:19.0550934Z (EngineCore_DP0 pid=4979) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:19.0551429Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0552073Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:19.0552661Z (EngineCore_DP0 pid=4979) return func(*args, **kwargs) 2025-10-10T01:45:19.0553045Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0553724Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:19.0554355Z (EngineCore_DP0 pid=4979) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:19.0554782Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0555451Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:19.0556110Z (EngineCore_DP0 pid=4979) module = importlib.import_module(module_name) 2025-10-10T01:45:19.0556555Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0557133Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:19.0557764Z (EngineCore_DP0 pid=4979) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:19.0558256Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0558754Z (EngineCore_DP0 pid=4979) File "", line 1387, in _gcd_import 2025-10-10T01:45:19.0559569Z (EngineCore_DP0 pid=4979) File "", line 1360, in _find_and_load 2025-10-10T01:45:19.0560267Z (EngineCore_DP0 pid=4979) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:19.0560960Z (EngineCore_DP0 pid=4979) File "", line 935, in _load_unlocked 2025-10-10T01:45:19.0561595Z (EngineCore_DP0 pid=4979) File "", line 999, in exec_module 2025-10-10T01:45:19.0562193Z (EngineCore_DP0 pid=4979) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:19.0562928Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:19.0563606Z (EngineCore_DP0 pid=4979) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:19.0564436Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:19.0565179Z (EngineCore_DP0 pid=4979) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:19.0565959Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:19.0566632Z (EngineCore_DP0 pid=4979) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:19.0567422Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:19.0568240Z (EngineCore_DP0 pid=4979) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:19.0568723Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0569419Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:19.0570105Z (EngineCore_DP0 pid=4979) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:19.0570546Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0571283Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:19.0572026Z (EngineCore_DP0 pid=4979) return _is_fa2_supported(device)[0] 2025-10-10T01:45:19.0572441Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0573151Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:19.0573859Z (EngineCore_DP0 pid=4979) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:19.0580905Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0581650Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:19.0582325Z (EngineCore_DP0 pid=4979) prop = get_device_properties(device) 2025-10-10T01:45:19.0582771Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:19.0583457Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:19.0584241Z (EngineCore_DP0 pid=4979) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:19.0584652Z (EngineCore_DP0 pid=4979) ^^^^^^^^^^^^ 2025-10-10T01:45:19.0585241Z (EngineCore_DP0 pid=4979) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:19.0585802Z (EngineCore_DP0 pid=4979) raise RuntimeError( 2025-10-10T01:45:19.0586500Z (EngineCore_DP0 pid=4979) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:19.4927947Z FAILED 2025-10-10T01:45:19.5058273Z models/test_initialization.py::test_can_initialize_large_subset[Gemma3nForConditionalGeneration] Fork a new process to run a test 4983 2025-10-10T01:45:19.5069383Z Fork a new process to run a test 0 2025-10-10T01:45:19.5343257Z INFO 10-10 01:45:19 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma3nForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/gemma-3n-E2B-it'} 2025-10-10T01:45:20.2555448Z INFO 10-10 01:45:20 [model.py:551] Resolved architecture: Gemma3nForConditionalGeneration 2025-10-10T01:45:20.2555987Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:45:20.2801936Z INFO 10-10 01:45:20 [model.py:1545] Using max model len 32768 2025-10-10T01:45:20.4629484Z INFO 10-10 01:45:20 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:45:23.6108779Z (EngineCore_DP0 pid=4990) INFO 10-10 01:45:23 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:45:23.6183330Z (EngineCore_DP0 pid=4990) INFO 10-10 01:45:23 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='google/gemma-3n-E2B-it', speculative_config=None, tokenizer='google/gemma-3n-E2B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=google/gemma-3n-E2B-it, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:23.7217126Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:23.7217819Z 2025-10-10T01:45:23.7218256Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:23.7218766Z 2025-10-10T01:45:23.7219760Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:23.7220424Z 2025-10-10T01:45:23.7220841Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:23.7221250Z 2025-10-10T01:45:23.7221597Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7221947Z 2025-10-10T01:45:23.7222896Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:23.7223532Z 2025-10-10T01:45:23.7223985Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:23.7224465Z 2025-10-10T01:45:23.7225001Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:23.7225472Z 2025-10-10T01:45:23.7225800Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:23.7226116Z 2025-10-10T01:45:23.7226408Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7226692Z 2025-10-10T01:45:23.7227207Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:23.7227681Z 2025-10-10T01:45:23.7227939Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] self._init_executor() 2025-10-10T01:45:23.7228368Z 2025-10-10T01:45:23.7229437Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:23.7230199Z 2025-10-10T01:45:23.7230756Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:23.7231473Z 2025-10-10T01:45:23.7232306Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:23.7232870Z 2025-10-10T01:45:23.7233246Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:23.7233713Z 2025-10-10T01:45:23.7234037Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7234323Z 2025-10-10T01:45:23.7234836Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:23.7235304Z 2025-10-10T01:45:23.7235573Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:23.7235853Z 2025-10-10T01:45:23.7236100Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7236487Z 2025-10-10T01:45:23.7237026Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:23.7237515Z 2025-10-10T01:45:23.7237806Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:23.7238101Z 2025-10-10T01:45:23.7238365Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7238632Z 2025-10-10T01:45:23.7239289Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:23.7239778Z 2025-10-10T01:45:23.7240091Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:23.7240389Z 2025-10-10T01:45:23.7241343Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7241649Z 2025-10-10T01:45:23.7242098Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:23.7242518Z 2025-10-10T01:45:23.7242856Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:23.7243190Z 2025-10-10T01:45:23.7243481Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7243767Z 2025-10-10T01:45:23.7244105Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:23.7244440Z 2025-10-10T01:45:23.7244785Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:23.7245119Z 2025-10-10T01:45:23.7245497Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:23.7245865Z 2025-10-10T01:45:23.7246216Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:23.7246549Z 2025-10-10T01:45:23.7246912Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:23.7247321Z 2025-10-10T01:45:23.7247695Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:23.7248061Z 2025-10-10T01:45:23.7248563Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:23.7249034Z 2025-10-10T01:45:23.7249383Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:23.7249726Z 2025-10-10T01:45:23.7250244Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:23.7250732Z 2025-10-10T01:45:23.7251101Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:23.7251460Z 2025-10-10T01:45:23.7252014Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:23.7252563Z 2025-10-10T01:45:23.7252862Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:23.7253152Z 2025-10-10T01:45:23.7253899Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:23.7254584Z 2025-10-10T01:45:23.7255018Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:23.7255431Z 2025-10-10T01:45:23.7255722Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7255989Z 2025-10-10T01:45:23.7256619Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:23.7257146Z 2025-10-10T01:45:23.7257441Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:23.7257741Z 2025-10-10T01:45:23.7258010Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7258281Z 2025-10-10T01:45:23.7258882Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:23.7259430Z 2025-10-10T01:45:23.7259713Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:23.7259996Z 2025-10-10T01:45:23.7260262Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7260531Z 2025-10-10T01:45:23.7261111Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:23.7261788Z 2025-10-10T01:45:23.7262107Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:23.7262425Z 2025-10-10T01:45:23.7262696Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7263078Z 2025-10-10T01:45:23.7263681Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:23.7264273Z 2025-10-10T01:45:23.7264605Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:23.7264947Z 2025-10-10T01:45:23.7265240Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7265508Z 2025-10-10T01:45:23.7266028Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:23.7266504Z 2025-10-10T01:45:23.7266817Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:23.7267120Z 2025-10-10T01:45:23.7267365Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:23.7267605Z 2025-10-10T01:45:23.7268153Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:23.7268626Z 2025-10-10T01:45:23.7268872Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] raise RuntimeError( 2025-10-10T01:45:23.7269132Z 2025-10-10T01:45:23.7269720Z (EngineCore_DP0 pid=4990) ERROR 10-10 01:45:23 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:23.7270414Z (EngineCore_DP0 pid=4990) Process EngineCore_DP0: 2025-10-10T01:45:23.7270807Z (EngineCore_DP0 pid=4990) Traceback (most recent call last): 2025-10-10T01:45:23.7271409Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:23.7271926Z (EngineCore_DP0 pid=4990) self.run() 2025-10-10T01:45:23.7272551Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:23.7273126Z (EngineCore_DP0 pid=4990) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:23.7273798Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:23.7274353Z (EngineCore_DP0 pid=4990) raise e 2025-10-10T01:45:23.7274938Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:23.7275584Z (EngineCore_DP0 pid=4990) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:23.7276036Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7276654Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:23.7277296Z (EngineCore_DP0 pid=4990) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:23.7277956Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:23.7278570Z (EngineCore_DP0 pid=4990) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:23.7279066Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7279784Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:23.7280372Z (EngineCore_DP0 pid=4990) self._init_executor() 2025-10-10T01:45:23.7281031Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:23.7281717Z (EngineCore_DP0 pid=4990) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:23.7282438Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:23.7283170Z (EngineCore_DP0 pid=4990) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:23.7283674Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7284316Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:23.7284898Z (EngineCore_DP0 pid=4990) return func(*args, **kwargs) 2025-10-10T01:45:23.7285324Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7285961Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:23.7286586Z (EngineCore_DP0 pid=4990) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:23.7287021Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7287681Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:23.7288349Z (EngineCore_DP0 pid=4990) module = importlib.import_module(module_name) 2025-10-10T01:45:23.7288790Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7289363Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:23.7290069Z (EngineCore_DP0 pid=4990) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:23.7290579Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7291071Z (EngineCore_DP0 pid=4990) File "", line 1387, in _gcd_import 2025-10-10T01:45:23.7291623Z (EngineCore_DP0 pid=4990) File "", line 1360, in _find_and_load 2025-10-10T01:45:23.7292206Z (EngineCore_DP0 pid=4990) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:23.7292781Z (EngineCore_DP0 pid=4990) File "", line 935, in _load_unlocked 2025-10-10T01:45:23.7293348Z (EngineCore_DP0 pid=4990) File "", line 999, in exec_module 2025-10-10T01:45:23.7293957Z (EngineCore_DP0 pid=4990) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:23.7294699Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:23.7295378Z (EngineCore_DP0 pid=4990) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:23.7296339Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:23.7297083Z (EngineCore_DP0 pid=4990) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:23.7297978Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:23.7298660Z (EngineCore_DP0 pid=4990) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:23.7299458Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:23.7300282Z (EngineCore_DP0 pid=4990) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:23.7300761Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7301453Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:23.7302147Z (EngineCore_DP0 pid=4990) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:23.7302739Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7303929Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:23.7304776Z (EngineCore_DP0 pid=4990) return _is_fa2_supported(device)[0] 2025-10-10T01:45:23.7305212Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7305929Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:23.7306646Z (EngineCore_DP0 pid=4990) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:23.7307106Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7307770Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:23.7308408Z (EngineCore_DP0 pid=4990) prop = get_device_properties(device) 2025-10-10T01:45:23.7308981Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:23.7309656Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:23.7310339Z (EngineCore_DP0 pid=4990) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:23.7310742Z (EngineCore_DP0 pid=4990) ^^^^^^^^^^^^ 2025-10-10T01:45:23.7311325Z (EngineCore_DP0 pid=4990) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:23.7311887Z (EngineCore_DP0 pid=4990) raise RuntimeError( 2025-10-10T01:45:23.7312572Z (EngineCore_DP0 pid=4990) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:24.1732327Z FAILED 2025-10-10T01:45:24.1868804Z models/test_initialization.py::test_can_initialize_large_subset[QwenVLForConditionalGeneration] Fork a new process to run a test 4994 2025-10-10T01:45:24.1879760Z Fork a new process to run a test 0 2025-10-10T01:45:24.2148856Z INFO 10-10 01:45:24 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='QwenVLForConditionalGeneration', exist_overrides={'architectures': ['QwenVLForConditionalGeneration']}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen-VL'} 2025-10-10T01:45:24.7607411Z 2025-10-10T01:45:24.7609439Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:45:24.7609774Z config.json: 1.16kB [00:00, 6.05MB/s] 2025-10-10T01:45:24.8480374Z 2025-10-10T01:45:24.8481585Z configuration_qwen.py: 0.00B [00:00, ?B/s] 2025-10-10T01:45:24.8481975Z configuration_qwen.py: 2.09kB [00:00, 17.0MB/s] 2025-10-10T01:45:24.8567771Z A new version of the following files was downloaded from https://huggingface.co/Qwen/Qwen-VL: 2025-10-10T01:45:24.8568255Z - configuration_qwen.py 2025-10-10T01:45:24.8568860Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:45:31.8527337Z INFO 10-10 01:45:31 [model.py:551] Resolved architecture: QwenVLForConditionalGeneration 2025-10-10T01:45:31.8527862Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:45:31.8775593Z INFO 10-10 01:45:31 [model.py:1545] Using max model len 2048 2025-10-10T01:45:31.8777903Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:45:31.9445758Z INFO 10-10 01:45:31 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:45:31.9845158Z 2025-10-10T01:45:31.9846314Z tokenizer_config.json: 0% 0.00/173 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:36.3728815Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:36.3729244Z 2025-10-10T01:45:36.3729725Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:36.3730198Z 2025-10-10T01:45:36.3730910Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:36.3731792Z 2025-10-10T01:45:36.3732186Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:36.3732594Z 2025-10-10T01:45:36.3732947Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3733297Z 2025-10-10T01:45:36.3733913Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:36.3734479Z 2025-10-10T01:45:36.3734898Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:36.3735500Z 2025-10-10T01:45:36.3736193Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:36.3736660Z 2025-10-10T01:45:36.3736998Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:36.3737304Z 2025-10-10T01:45:36.3737590Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3737861Z 2025-10-10T01:45:36.3738419Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:36.3739196Z 2025-10-10T01:45:36.3739538Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] self._init_executor() 2025-10-10T01:45:36.3739980Z 2025-10-10T01:45:36.3740705Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:36.3741541Z 2025-10-10T01:45:36.3742089Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:36.3742621Z 2025-10-10T01:45:36.3743261Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:36.3743781Z 2025-10-10T01:45:36.3744248Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:36.3744594Z 2025-10-10T01:45:36.3744902Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3745201Z 2025-10-10T01:45:36.3745740Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:36.3746201Z 2025-10-10T01:45:36.3746479Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:36.3746757Z 2025-10-10T01:45:36.3747007Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3747270Z 2025-10-10T01:45:36.3747799Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:36.3748295Z 2025-10-10T01:45:36.3748615Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:36.3748951Z 2025-10-10T01:45:36.3749225Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3749492Z 2025-10-10T01:45:36.3750020Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:36.3750508Z 2025-10-10T01:45:36.3750807Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:36.3751107Z 2025-10-10T01:45:36.3751375Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3751665Z 2025-10-10T01:45:36.3752106Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:36.3752521Z 2025-10-10T01:45:36.3752941Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:36.3753270Z 2025-10-10T01:45:36.3753560Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3753836Z 2025-10-10T01:45:36.3754179Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:36.3754504Z 2025-10-10T01:45:36.3754856Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:36.3755189Z 2025-10-10T01:45:36.3755557Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:36.3755922Z 2025-10-10T01:45:36.3756262Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:36.3756594Z 2025-10-10T01:45:36.3756951Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:36.3757300Z 2025-10-10T01:45:36.3757680Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:36.3758046Z 2025-10-10T01:45:36.3758533Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:36.3759168Z 2025-10-10T01:45:36.3759530Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:36.3759870Z 2025-10-10T01:45:36.3760394Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:36.3760868Z 2025-10-10T01:45:36.3761244Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:36.3761607Z 2025-10-10T01:45:36.3762158Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:36.3762659Z 2025-10-10T01:45:36.3762947Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:36.3763244Z 2025-10-10T01:45:36.3763882Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:36.3764506Z 2025-10-10T01:45:36.3764871Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:36.3765223Z 2025-10-10T01:45:36.3765473Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3765735Z 2025-10-10T01:45:36.3766289Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:36.3766800Z 2025-10-10T01:45:36.3767101Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:36.3767391Z 2025-10-10T01:45:36.3767663Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3768000Z 2025-10-10T01:45:36.3768634Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:36.3769184Z 2025-10-10T01:45:36.3769463Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:36.3769756Z 2025-10-10T01:45:36.3770012Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3770286Z 2025-10-10T01:45:36.3770872Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:36.3771401Z 2025-10-10T01:45:36.3771720Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:36.3772035Z 2025-10-10T01:45:36.3772304Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3772572Z 2025-10-10T01:45:36.3773108Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:36.3773587Z 2025-10-10T01:45:36.3773913Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:36.3774196Z 2025-10-10T01:45:36.3774458Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3774724Z 2025-10-10T01:45:36.3775242Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:36.3775725Z 2025-10-10T01:45:36.3776032Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:36.3776336Z 2025-10-10T01:45:36.3776558Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:36.3776800Z 2025-10-10T01:45:36.3777280Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:36.3777732Z 2025-10-10T01:45:36.3777975Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] raise RuntimeError( 2025-10-10T01:45:36.3778227Z 2025-10-10T01:45:36.3778799Z (EngineCore_DP0 pid=5052) ERROR 10-10 01:45:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:36.3779584Z (EngineCore_DP0 pid=5052) Process EngineCore_DP0: 2025-10-10T01:45:36.3779972Z (EngineCore_DP0 pid=5052) Traceback (most recent call last): 2025-10-10T01:45:36.3780574Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:36.3781095Z (EngineCore_DP0 pid=5052) self.run() 2025-10-10T01:45:36.3781618Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:36.3782178Z (EngineCore_DP0 pid=5052) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:36.3782830Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:36.3783457Z (EngineCore_DP0 pid=5052) raise e 2025-10-10T01:45:36.3784048Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:36.3784681Z (EngineCore_DP0 pid=5052) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:36.3785115Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3785735Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:36.3786371Z (EngineCore_DP0 pid=5052) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:36.3787031Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:36.3787651Z (EngineCore_DP0 pid=5052) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:36.3788094Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3788739Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:36.3789317Z (EngineCore_DP0 pid=5052) self._init_executor() 2025-10-10T01:45:36.3789973Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:36.3790702Z (EngineCore_DP0 pid=5052) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:36.3791418Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:36.3792130Z (EngineCore_DP0 pid=5052) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:36.3792642Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3793301Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:36.3793891Z (EngineCore_DP0 pid=5052) return func(*args, **kwargs) 2025-10-10T01:45:36.3794266Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3794903Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:36.3795537Z (EngineCore_DP0 pid=5052) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:36.3795958Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3796942Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:36.3797594Z (EngineCore_DP0 pid=5052) module = importlib.import_module(module_name) 2025-10-10T01:45:36.3798041Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3798619Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:36.3799289Z (EngineCore_DP0 pid=5052) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:36.3799780Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3800282Z (EngineCore_DP0 pid=5052) File "", line 1387, in _gcd_import 2025-10-10T01:45:36.3800977Z (EngineCore_DP0 pid=5052) File "", line 1360, in _find_and_load 2025-10-10T01:45:36.3801591Z (EngineCore_DP0 pid=5052) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:36.3802168Z (EngineCore_DP0 pid=5052) File "", line 935, in _load_unlocked 2025-10-10T01:45:36.3802735Z (EngineCore_DP0 pid=5052) File "", line 999, in exec_module 2025-10-10T01:45:36.3803345Z (EngineCore_DP0 pid=5052) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:36.3804074Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:36.3804747Z (EngineCore_DP0 pid=5052) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:36.3805463Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:36.3806206Z (EngineCore_DP0 pid=5052) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:36.3807016Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:36.3807686Z (EngineCore_DP0 pid=5052) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:36.3808556Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:36.3809375Z (EngineCore_DP0 pid=5052) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:36.3809861Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3810559Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:36.3811247Z (EngineCore_DP0 pid=5052) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:36.3811681Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3812427Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:36.3813131Z (EngineCore_DP0 pid=5052) return _is_fa2_supported(device)[0] 2025-10-10T01:45:36.3813543Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3814258Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:36.3815037Z (EngineCore_DP0 pid=5052) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:36.3815494Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3816163Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:36.3816806Z (EngineCore_DP0 pid=5052) prop = get_device_properties(device) 2025-10-10T01:45:36.3817238Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:36.3817888Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:36.3818551Z (EngineCore_DP0 pid=5052) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:36.3819022Z (EngineCore_DP0 pid=5052) ^^^^^^^^^^^^ 2025-10-10T01:45:36.3819613Z (EngineCore_DP0 pid=5052) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:36.3820187Z (EngineCore_DP0 pid=5052) raise RuntimeError( 2025-10-10T01:45:36.3820872Z (EngineCore_DP0 pid=5052) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:36.8001250Z FAILED 2025-10-10T01:45:36.8132244Z models/test_initialization.py::test_can_initialize_large_subset[LlamaForCausalLM] Fork a new process to run a test 5056 2025-10-10T01:45:36.8142762Z Fork a new process to run a test 0 2025-10-10T01:45:36.8419208Z INFO 10-10 01:45:36 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlamaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'meta-llama/Llama-3.2-1B-Instruct'} 2025-10-10T01:45:36.9932712Z INFO 10-10 01:45:36 [model.py:551] Resolved architecture: LlamaForCausalLM 2025-10-10T01:45:36.9933173Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:45:37.0184544Z INFO 10-10 01:45:37 [model.py:1545] Using max model len 131072 2025-10-10T01:45:37.2328343Z INFO 10-10 01:45:37 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:45:37.7656520Z (EngineCore_DP0 pid=5063) INFO 10-10 01:45:37 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:45:37.7731531Z (EngineCore_DP0 pid=5063) INFO 10-10 01:45:37 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='meta-llama/Llama-3.2-1B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.2-1B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=meta-llama/Llama-3.2-1B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:37.9063850Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:37.9064402Z 2025-10-10T01:45:37.9065103Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:37.9065506Z 2025-10-10T01:45:37.9066215Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:37.9066833Z 2025-10-10T01:45:37.9067160Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:37.9067481Z 2025-10-10T01:45:37.9067761Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9068035Z 2025-10-10T01:45:37.9068518Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:37.9068975Z 2025-10-10T01:45:37.9069310Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:37.9069633Z 2025-10-10T01:45:37.9070369Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:37.9070826Z 2025-10-10T01:45:37.9071155Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:37.9071616Z 2025-10-10T01:45:37.9071899Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9072172Z 2025-10-10T01:45:37.9072684Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:37.9073156Z 2025-10-10T01:45:37.9073410Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] self._init_executor() 2025-10-10T01:45:37.9073671Z 2025-10-10T01:45:37.9074521Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:37.9075393Z 2025-10-10T01:45:37.9075972Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:37.9076568Z 2025-10-10T01:45:37.9077618Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:37.9078328Z 2025-10-10T01:45:37.9078713Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:37.9079174Z 2025-10-10T01:45:37.9079522Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9080011Z 2025-10-10T01:45:37.9080887Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:37.9081696Z 2025-10-10T01:45:37.9082143Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:37.9082595Z 2025-10-10T01:45:37.9082992Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9083398Z 2025-10-10T01:45:37.9084315Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:37.9085071Z 2025-10-10T01:45:37.9085526Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:37.9085979Z 2025-10-10T01:45:37.9086406Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9086833Z 2025-10-10T01:45:37.9087654Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:37.9088427Z 2025-10-10T01:45:37.9088904Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:37.9089390Z 2025-10-10T01:45:37.9089829Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9090292Z 2025-10-10T01:45:37.9091048Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:37.9091747Z 2025-10-10T01:45:37.9092331Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:37.9092892Z 2025-10-10T01:45:37.9093416Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9093982Z 2025-10-10T01:45:37.9094617Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:37.9095249Z 2025-10-10T01:45:37.9095853Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:37.9096670Z 2025-10-10T01:45:37.9097315Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:37.9097936Z 2025-10-10T01:45:37.9098523Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:37.9099098Z 2025-10-10T01:45:37.9099714Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:37.9100333Z 2025-10-10T01:45:37.9101005Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:37.9101643Z 2025-10-10T01:45:37.9102640Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:37.9103461Z 2025-10-10T01:45:37.9104093Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:37.9104720Z 2025-10-10T01:45:37.9105672Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:37.9106545Z 2025-10-10T01:45:37.9107205Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:37.9107838Z 2025-10-10T01:45:37.9108796Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:37.9109686Z 2025-10-10T01:45:37.9110321Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:37.9110847Z 2025-10-10T01:45:37.9111933Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:37.9112903Z 2025-10-10T01:45:37.9113525Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:37.9114127Z 2025-10-10T01:45:37.9114553Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9114998Z 2025-10-10T01:45:37.9116052Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:37.9117023Z 2025-10-10T01:45:37.9117574Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:37.9118124Z 2025-10-10T01:45:37.9118633Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9119197Z 2025-10-10T01:45:37.9120197Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:37.9121230Z 2025-10-10T01:45:37.9121711Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:37.9122186Z 2025-10-10T01:45:37.9122626Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9123066Z 2025-10-10T01:45:37.9124073Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:37.9124991Z 2025-10-10T01:45:37.9125540Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:37.9126058Z 2025-10-10T01:45:37.9126523Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9126977Z 2025-10-10T01:45:37.9127897Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:37.9128843Z 2025-10-10T01:45:37.9129374Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:37.9129910Z 2025-10-10T01:45:37.9130366Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9130823Z 2025-10-10T01:45:37.9131708Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:37.9132554Z 2025-10-10T01:45:37.9133078Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:37.9133592Z 2025-10-10T01:45:37.9133966Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:37.9134364Z 2025-10-10T01:45:37.9135190Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:37.9136132Z 2025-10-10T01:45:37.9136566Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] raise RuntimeError( 2025-10-10T01:45:37.9137001Z 2025-10-10T01:45:37.9137992Z (EngineCore_DP0 pid=5063) ERROR 10-10 01:45:37 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:37.9139180Z (EngineCore_DP0 pid=5063) Process EngineCore_DP0: 2025-10-10T01:45:37.9139919Z (EngineCore_DP0 pid=5063) Traceback (most recent call last): 2025-10-10T01:45:37.9141046Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:37.9141999Z (EngineCore_DP0 pid=5063) self.run() 2025-10-10T01:45:37.9142880Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:37.9143840Z (EngineCore_DP0 pid=5063) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:37.9144977Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:37.9145886Z (EngineCore_DP0 pid=5063) raise e 2025-10-10T01:45:37.9146865Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:37.9148009Z (EngineCore_DP0 pid=5063) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:37.9148764Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9149825Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:37.9150942Z (EngineCore_DP0 pid=5063) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:37.9152115Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:37.9153242Z (EngineCore_DP0 pid=5063) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:37.9154084Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9155168Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:37.9156138Z (EngineCore_DP0 pid=5063) self._init_executor() 2025-10-10T01:45:37.9157280Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:37.9158558Z (EngineCore_DP0 pid=5063) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:37.9159915Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:37.9161188Z (EngineCore_DP0 pid=5063) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:37.9162070Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9163187Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:37.9164201Z (EngineCore_DP0 pid=5063) return func(*args, **kwargs) 2025-10-10T01:45:37.9164887Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9166154Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:37.9167305Z (EngineCore_DP0 pid=5063) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:37.9168057Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9169283Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:37.9170491Z (EngineCore_DP0 pid=5063) module = importlib.import_module(module_name) 2025-10-10T01:45:37.9171290Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9172343Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:37.9173507Z (EngineCore_DP0 pid=5063) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:37.9174412Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9175292Z (EngineCore_DP0 pid=5063) File "", line 1387, in _gcd_import 2025-10-10T01:45:37.9176287Z (EngineCore_DP0 pid=5063) File "", line 1360, in _find_and_load 2025-10-10T01:45:37.9177300Z (EngineCore_DP0 pid=5063) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:37.9178348Z (EngineCore_DP0 pid=5063) File "", line 935, in _load_unlocked 2025-10-10T01:45:37.9179385Z (EngineCore_DP0 pid=5063) File "", line 999, in exec_module 2025-10-10T01:45:37.9180494Z (EngineCore_DP0 pid=5063) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:37.9181799Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:37.9183054Z (EngineCore_DP0 pid=5063) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:37.9184284Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:37.9185448Z (EngineCore_DP0 pid=5063) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:37.9186797Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:37.9187981Z (EngineCore_DP0 pid=5063) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:37.9189332Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:37.9190664Z (EngineCore_DP0 pid=5063) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:37.9191431Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9192554Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:37.9193677Z (EngineCore_DP0 pid=5063) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:37.9194407Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9195667Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:37.9197298Z (EngineCore_DP0 pid=5063) return _is_fa2_supported(device)[0] 2025-10-10T01:45:37.9198030Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9199286Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:37.9200454Z (EngineCore_DP0 pid=5063) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:37.9201203Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9202289Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:37.9203329Z (EngineCore_DP0 pid=5063) prop = get_device_properties(device) 2025-10-10T01:45:37.9204005Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:37.9205068Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:37.9206164Z (EngineCore_DP0 pid=5063) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:37.9206831Z (EngineCore_DP0 pid=5063) ^^^^^^^^^^^^ 2025-10-10T01:45:37.9207775Z (EngineCore_DP0 pid=5063) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:37.9208773Z (EngineCore_DP0 pid=5063) raise RuntimeError( 2025-10-10T01:45:37.9210027Z (EngineCore_DP0 pid=5063) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:38.3252649Z FAILED 2025-10-10T01:45:38.3383266Z models/test_initialization.py::test_can_initialize_large_subset[Ovis] Fork a new process to run a test 5067 2025-10-10T01:45:38.3395306Z Fork a new process to run a test 0 2025-10-10T01:45:38.3399713Z `transformers==4.56.2` installed, but `transformers<=4.53` is required to run this model. Reason: HF model is not compatible 2025-10-10T01:45:38.6436933Z PASSED 2025-10-10T01:45:38.6568374Z models/test_initialization.py::test_can_initialize_large_subset[MedusaModel] Fork a new process to run a test 5068 2025-10-10T01:45:38.6580414Z Fork a new process to run a test 0 2025-10-10T01:45:38.6866078Z INFO 10-10 01:45:38 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MedusaModel', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'abhigoyal/vllm-medusa-llama-68m-random', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'JackFram/llama-68m'} 2025-10-10T01:45:38.8035838Z 2025-10-10T01:45:38.8037196Z config.json: 0% 0.00/545 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message 2025-10-10T01:45:46.6427850Z You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. 2025-10-10T01:45:46.8776143Z 2025-10-10T01:45:46.8776877Z generation_config.json: 0% 0.00/137 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:47.0321810Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:47.0322546Z 2025-10-10T01:45:47.0323213Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:47.0323611Z 2025-10-10T01:45:47.0324539Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:47.0325210Z 2025-10-10T01:45:47.0325559Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:47.0325867Z 2025-10-10T01:45:47.0326144Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0326417Z 2025-10-10T01:45:47.0326908Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:47.0327573Z 2025-10-10T01:45:47.0327916Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:47.0328235Z 2025-10-10T01:45:47.0329038Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:47.0329723Z 2025-10-10T01:45:47.0330098Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:47.0330405Z 2025-10-10T01:45:47.0330693Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0330972Z 2025-10-10T01:45:47.0331476Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:47.0331978Z 2025-10-10T01:45:47.0332459Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] self._init_executor() 2025-10-10T01:45:47.0332912Z 2025-10-10T01:45:47.0333531Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:47.0334454Z 2025-10-10T01:45:47.0335070Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:47.0335517Z 2025-10-10T01:45:47.0336178Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:47.0337109Z 2025-10-10T01:45:47.0337561Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:47.0337906Z 2025-10-10T01:45:47.0338200Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0338482Z 2025-10-10T01:45:47.0339113Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:47.0339599Z 2025-10-10T01:45:47.0339882Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:47.0340151Z 2025-10-10T01:45:47.0340399Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0340662Z 2025-10-10T01:45:47.0341176Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:47.0341656Z 2025-10-10T01:45:47.0341941Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:47.0342232Z 2025-10-10T01:45:47.0342502Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0342767Z 2025-10-10T01:45:47.0343296Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:47.0343773Z 2025-10-10T01:45:47.0344085Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:47.0344381Z 2025-10-10T01:45:47.0344660Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0345012Z 2025-10-10T01:45:47.0345463Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:47.0345882Z 2025-10-10T01:45:47.0346339Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:47.0346708Z 2025-10-10T01:45:47.0347006Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0347294Z 2025-10-10T01:45:47.0347639Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:47.0347977Z 2025-10-10T01:45:47.0348325Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:47.0348663Z 2025-10-10T01:45:47.0349049Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:47.0349467Z 2025-10-10T01:45:47.0349830Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:47.0350160Z 2025-10-10T01:45:47.0350533Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:47.0350884Z 2025-10-10T01:45:47.0351258Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:47.0351622Z 2025-10-10T01:45:47.0352123Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:47.0352592Z 2025-10-10T01:45:47.0352939Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:47.0353294Z 2025-10-10T01:45:47.0353865Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:47.0354355Z 2025-10-10T01:45:47.0354735Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:47.0355097Z 2025-10-10T01:45:47.0355644Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:47.0356151Z 2025-10-10T01:45:47.0356450Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:47.0356743Z 2025-10-10T01:45:47.0357377Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:47.0357958Z 2025-10-10T01:45:47.0358330Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:47.0358678Z 2025-10-10T01:45:47.0358933Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0359339Z 2025-10-10T01:45:47.0359900Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:47.0360481Z 2025-10-10T01:45:47.0360779Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:47.0361078Z 2025-10-10T01:45:47.0361406Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0361692Z 2025-10-10T01:45:47.0362300Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:47.0362847Z 2025-10-10T01:45:47.0363133Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:47.0363412Z 2025-10-10T01:45:47.0363674Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0363940Z 2025-10-10T01:45:47.0364517Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:47.0365102Z 2025-10-10T01:45:47.0365422Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:47.0365735Z 2025-10-10T01:45:47.0366008Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0366282Z 2025-10-10T01:45:47.0366802Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:47.0367280Z 2025-10-10T01:45:47.0367555Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:47.0367843Z 2025-10-10T01:45:47.0368100Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0368364Z 2025-10-10T01:45:47.0368944Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:47.0369419Z 2025-10-10T01:45:47.0369734Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:47.0370032Z 2025-10-10T01:45:47.0370276Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:47.0370512Z 2025-10-10T01:45:47.0370989Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:47.0371438Z 2025-10-10T01:45:47.0371681Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] raise RuntimeError( 2025-10-10T01:45:47.0371939Z 2025-10-10T01:45:47.0372507Z (EngineCore_DP0 pid=5146) ERROR 10-10 01:45:47 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:47.0373207Z (EngineCore_DP0 pid=5146) Process EngineCore_DP0: 2025-10-10T01:45:47.0373599Z (EngineCore_DP0 pid=5146) Traceback (most recent call last): 2025-10-10T01:45:47.0374219Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:47.0374731Z (EngineCore_DP0 pid=5146) self.run() 2025-10-10T01:45:47.0375250Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:47.0375866Z (EngineCore_DP0 pid=5146) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:47.0376585Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:47.0377153Z (EngineCore_DP0 pid=5146) raise e 2025-10-10T01:45:47.0377749Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:47.0378375Z (EngineCore_DP0 pid=5146) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:47.0378806Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0379421Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:47.0380058Z (EngineCore_DP0 pid=5146) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:47.0380715Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:47.0381391Z (EngineCore_DP0 pid=5146) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:47.0381846Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0382493Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:47.0383077Z (EngineCore_DP0 pid=5146) self._init_executor() 2025-10-10T01:45:47.0383737Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:47.0384430Z (EngineCore_DP0 pid=5146) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:47.0385145Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:47.0385882Z (EngineCore_DP0 pid=5146) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:47.0386419Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0387076Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:47.0387665Z (EngineCore_DP0 pid=5146) return func(*args, **kwargs) 2025-10-10T01:45:47.0388043Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0388684Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:47.0389312Z (EngineCore_DP0 pid=5146) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:47.0389744Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0390415Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:47.0391056Z (EngineCore_DP0 pid=5146) module = importlib.import_module(module_name) 2025-10-10T01:45:47.0391495Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0392072Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:47.0392695Z (EngineCore_DP0 pid=5146) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:47.0393244Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0393749Z (EngineCore_DP0 pid=5146) File "", line 1387, in _gcd_import 2025-10-10T01:45:47.0394355Z (EngineCore_DP0 pid=5146) File "", line 1360, in _find_and_load 2025-10-10T01:45:47.0394948Z (EngineCore_DP0 pid=5146) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:47.0395523Z (EngineCore_DP0 pid=5146) File "", line 935, in _load_unlocked 2025-10-10T01:45:47.0396349Z (EngineCore_DP0 pid=5146) File "", line 999, in exec_module 2025-10-10T01:45:47.0397061Z (EngineCore_DP0 pid=5146) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:47.0397800Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:47.0398473Z (EngineCore_DP0 pid=5146) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:47.0399405Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:47.0400123Z (EngineCore_DP0 pid=5146) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:47.0400910Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:47.0401566Z (EngineCore_DP0 pid=5146) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:47.0402353Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:47.0403166Z (EngineCore_DP0 pid=5146) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:47.0403651Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0404427Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:47.0405118Z (EngineCore_DP0 pid=5146) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:47.0405543Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0406279Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:47.0406991Z (EngineCore_DP0 pid=5146) return _is_fa2_supported(device)[0] 2025-10-10T01:45:47.0407402Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0408111Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:47.0408818Z (EngineCore_DP0 pid=5146) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:47.0409280Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0409946Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:47.0410592Z (EngineCore_DP0 pid=5146) prop = get_device_properties(device) 2025-10-10T01:45:47.0411010Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:47.0411784Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:47.0412528Z (EngineCore_DP0 pid=5146) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:47.0412949Z (EngineCore_DP0 pid=5146) ^^^^^^^^^^^^ 2025-10-10T01:45:47.0413529Z (EngineCore_DP0 pid=5146) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:47.0414093Z (EngineCore_DP0 pid=5146) raise RuntimeError( 2025-10-10T01:45:47.0414776Z (EngineCore_DP0 pid=5146) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:47.4405296Z FAILED 2025-10-10T01:45:47.4538261Z models/test_initialization.py::test_can_initialize_large_subset[Glm4vForConditionalGeneration] Fork a new process to run a test 5150 2025-10-10T01:45:47.4548108Z Fork a new process to run a test 0 2025-10-10T01:45:47.4824787Z INFO 10-10 01:45:47 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Glm4vForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'zai-org/GLM-4.1V-9B-Thinking'} 2025-10-10T01:45:47.5645635Z 2025-10-10T01:45:47.5647872Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:45:47.5648496Z config.json: 1.39kB [00:00, 6.40MB/s] 2025-10-10T01:45:47.6637597Z 2025-10-10T01:45:47.6638941Z preprocessor_config.json: 0% 0.00/364 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:56.9036034Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:56.9036704Z 2025-10-10T01:45:56.9037339Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:56.9037973Z 2025-10-10T01:45:56.9039283Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:56.9039946Z 2025-10-10T01:45:56.9040351Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:56.9040725Z 2025-10-10T01:45:56.9041089Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9041437Z 2025-10-10T01:45:56.9042296Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:56.9042867Z 2025-10-10T01:45:56.9043281Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:56.9043693Z 2025-10-10T01:45:56.9044604Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:56.9045285Z 2025-10-10T01:45:56.9045620Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:56.9045935Z 2025-10-10T01:45:56.9046223Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9046505Z 2025-10-10T01:45:56.9047021Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:56.9047850Z 2025-10-10T01:45:56.9048305Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] self._init_executor() 2025-10-10T01:45:56.9048759Z 2025-10-10T01:45:56.9049722Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:56.9050781Z 2025-10-10T01:45:56.9051368Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:56.9051725Z 2025-10-10T01:45:56.9052394Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:56.9052908Z 2025-10-10T01:45:56.9053313Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:56.9053647Z 2025-10-10T01:45:56.9053942Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9054230Z 2025-10-10T01:45:56.9054732Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:56.9055190Z 2025-10-10T01:45:56.9055455Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:56.9055742Z 2025-10-10T01:45:56.9056105Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9056372Z 2025-10-10T01:45:56.9056887Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:56.9057363Z 2025-10-10T01:45:56.9057646Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:56.9057931Z 2025-10-10T01:45:56.9058202Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9058472Z 2025-10-10T01:45:56.9059001Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:56.9059484Z 2025-10-10T01:45:56.9059792Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:56.9060143Z 2025-10-10T01:45:56.9060431Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9060703Z 2025-10-10T01:45:56.9061138Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:56.9061550Z 2025-10-10T01:45:56.9061885Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:56.9062216Z 2025-10-10T01:45:56.9062500Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9062782Z 2025-10-10T01:45:56.9063124Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:56.9063456Z 2025-10-10T01:45:56.9063826Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:56.9064166Z 2025-10-10T01:45:56.9064545Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:56.9064901Z 2025-10-10T01:45:56.9065252Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:56.9065643Z 2025-10-10T01:45:56.9066015Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:56.9066364Z 2025-10-10T01:45:56.9066786Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:56.9067158Z 2025-10-10T01:45:56.9067670Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:56.9068133Z 2025-10-10T01:45:56.9068493Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:56.9068842Z 2025-10-10T01:45:56.9069369Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:56.9069859Z 2025-10-10T01:45:56.9070229Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:56.9070650Z 2025-10-10T01:45:56.9071225Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:56.9071741Z 2025-10-10T01:45:56.9072045Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:56.9072337Z 2025-10-10T01:45:56.9072980Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:56.9073555Z 2025-10-10T01:45:56.9073935Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:56.9074284Z 2025-10-10T01:45:56.9074549Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9074822Z 2025-10-10T01:45:56.9075436Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:56.9075981Z 2025-10-10T01:45:56.9076282Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:56.9076583Z 2025-10-10T01:45:56.9076852Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9077133Z 2025-10-10T01:45:56.9077728Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:56.9078274Z 2025-10-10T01:45:56.9078564Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:56.9078849Z 2025-10-10T01:45:56.9079220Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9079487Z 2025-10-10T01:45:56.9080073Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:56.9080631Z 2025-10-10T01:45:56.9080954Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:56.9081341Z 2025-10-10T01:45:56.9081620Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9081902Z 2025-10-10T01:45:56.9082518Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:56.9083016Z 2025-10-10T01:45:56.9083299Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:56.9083592Z 2025-10-10T01:45:56.9083860Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9084132Z 2025-10-10T01:45:56.9084656Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:56.9085132Z 2025-10-10T01:45:56.9085442Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:56.9085741Z 2025-10-10T01:45:56.9086039Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:56.9086292Z 2025-10-10T01:45:56.9086795Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:56.9087257Z 2025-10-10T01:45:56.9087497Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] raise RuntimeError( 2025-10-10T01:45:56.9087762Z 2025-10-10T01:45:56.9088324Z (EngineCore_DP0 pid=5233) ERROR 10-10 01:45:56 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:56.9089022Z (EngineCore_DP0 pid=5233) Process EngineCore_DP0: 2025-10-10T01:45:56.9089417Z (EngineCore_DP0 pid=5233) Traceback (most recent call last): 2025-10-10T01:45:56.9090032Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:56.9090597Z (EngineCore_DP0 pid=5233) self.run() 2025-10-10T01:45:56.9091131Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:56.9091692Z (EngineCore_DP0 pid=5233) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:56.9092365Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:56.9092910Z (EngineCore_DP0 pid=5233) raise e 2025-10-10T01:45:56.9093497Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:56.9094136Z (EngineCore_DP0 pid=5233) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:56.9094598Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9095220Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:56.9095873Z (EngineCore_DP0 pid=5233) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:56.9096916Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:56.9097558Z (EngineCore_DP0 pid=5233) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:56.9098141Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9098833Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:56.9099502Z (EngineCore_DP0 pid=5233) self._init_executor() 2025-10-10T01:45:56.9100192Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:56.9100881Z (EngineCore_DP0 pid=5233) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:56.9101630Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:56.9102347Z (EngineCore_DP0 pid=5233) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:56.9102849Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9103507Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:56.9104186Z (EngineCore_DP0 pid=5233) return func(*args, **kwargs) 2025-10-10T01:45:56.9104585Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9105241Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:56.9105874Z (EngineCore_DP0 pid=5233) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:56.9106299Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9106962Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:56.9107621Z (EngineCore_DP0 pid=5233) module = importlib.import_module(module_name) 2025-10-10T01:45:56.9108062Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9108726Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:56.9109367Z (EngineCore_DP0 pid=5233) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:56.9109856Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9110350Z (EngineCore_DP0 pid=5233) File "", line 1387, in _gcd_import 2025-10-10T01:45:56.9110914Z (EngineCore_DP0 pid=5233) File "", line 1360, in _find_and_load 2025-10-10T01:45:56.9111507Z (EngineCore_DP0 pid=5233) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:56.9112084Z (EngineCore_DP0 pid=5233) File "", line 935, in _load_unlocked 2025-10-10T01:45:56.9112652Z (EngineCore_DP0 pid=5233) File "", line 999, in exec_module 2025-10-10T01:45:56.9113253Z (EngineCore_DP0 pid=5233) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:56.9113990Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:56.9114676Z (EngineCore_DP0 pid=5233) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:56.9115883Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:56.9116917Z (EngineCore_DP0 pid=5233) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:56.9117791Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:56.9118493Z (EngineCore_DP0 pid=5233) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:56.9119389Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:56.9120212Z (EngineCore_DP0 pid=5233) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:56.9120692Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9121380Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:56.9122066Z (EngineCore_DP0 pid=5233) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:56.9122506Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9123318Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:56.9124018Z (EngineCore_DP0 pid=5233) return _is_fa2_supported(device)[0] 2025-10-10T01:45:56.9124443Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9125161Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:56.9125881Z (EngineCore_DP0 pid=5233) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:56.9126338Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9127007Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:56.9127691Z (EngineCore_DP0 pid=5233) prop = get_device_properties(device) 2025-10-10T01:45:56.9128106Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:56.9128765Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:56.9129423Z (EngineCore_DP0 pid=5233) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:56.9129830Z (EngineCore_DP0 pid=5233) ^^^^^^^^^^^^ 2025-10-10T01:45:56.9130412Z (EngineCore_DP0 pid=5233) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:56.9130977Z (EngineCore_DP0 pid=5233) raise RuntimeError( 2025-10-10T01:45:56.9131668Z (EngineCore_DP0 pid=5233) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:57.3354892Z FAILED 2025-10-10T01:45:57.3486585Z models/test_initialization.py::test_can_initialize_large_subset[InternLMForCausalLM] Fork a new process to run a test 5237 2025-10-10T01:45:57.3496526Z Fork a new process to run a test 0 2025-10-10T01:45:57.3771136Z INFO 10-10 01:45:57 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternLMForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'internlm/internlm-chat-7b'} 2025-10-10T01:45:57.5166981Z 2025-10-10T01:45:57.5167963Z config.json: 0% 0.00/731 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:45:59.0256092Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] EngineCore failed to start. 2025-10-10T01:45:59.0256600Z 2025-10-10T01:45:59.0256944Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] Traceback (most recent call last): 2025-10-10T01:45:59.0257266Z 2025-10-10T01:45:59.0258218Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:59.0258807Z 2025-10-10T01:45:59.0259122Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:59.0259443Z 2025-10-10T01:45:59.0259718Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0259997Z 2025-10-10T01:45:59.0260487Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:59.0260944Z 2025-10-10T01:45:59.0261545Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:59.0261883Z 2025-10-10T01:45:59.0262415Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:59.0262954Z 2025-10-10T01:45:59.0263529Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:59.0264068Z 2025-10-10T01:45:59.0264366Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0264652Z 2025-10-10T01:45:59.0265180Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:59.0265677Z 2025-10-10T01:45:59.0265935Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] self._init_executor() 2025-10-10T01:45:59.0266201Z 2025-10-10T01:45:59.0266858Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:59.0267746Z 2025-10-10T01:45:59.0268321Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:59.0269040Z 2025-10-10T01:45:59.0269992Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:59.0270881Z 2025-10-10T01:45:59.0271607Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:59.0272217Z 2025-10-10T01:45:59.0272529Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0272813Z 2025-10-10T01:45:59.0273321Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:59.0273791Z 2025-10-10T01:45:59.0274067Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:45:59.0274348Z 2025-10-10T01:45:59.0274602Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0274860Z 2025-10-10T01:45:59.0275387Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:59.0275978Z 2025-10-10T01:45:59.0276270Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:59.0276564Z 2025-10-10T01:45:59.0276838Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0277114Z 2025-10-10T01:45:59.0277656Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:59.0278156Z 2025-10-10T01:45:59.0278460Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:45:59.0278758Z 2025-10-10T01:45:59.0279195Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0279469Z 2025-10-10T01:45:59.0279974Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:59.0280387Z 2025-10-10T01:45:59.0280732Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:59.0281057Z 2025-10-10T01:45:59.0281339Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0281622Z 2025-10-10T01:45:59.0281969Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:45:59.0282302Z 2025-10-10T01:45:59.0282651Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:45:59.0282993Z 2025-10-10T01:45:59.0283371Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:59.0283733Z 2025-10-10T01:45:59.0284071Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:45:59.0284401Z 2025-10-10T01:45:59.0284765Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:45:59.0285161Z 2025-10-10T01:45:59.0285536Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:59.0285900Z 2025-10-10T01:45:59.0286454Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:59.0286925Z 2025-10-10T01:45:59.0287273Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:59.0287609Z 2025-10-10T01:45:59.0288121Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:59.0288598Z 2025-10-10T01:45:59.0288966Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:59.0289334Z 2025-10-10T01:45:59.0289883Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:59.0290458Z 2025-10-10T01:45:59.0290756Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:45:59.0291064Z 2025-10-10T01:45:59.0291715Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:59.0292290Z 2025-10-10T01:45:59.0292659Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:59.0293012Z 2025-10-10T01:45:59.0293268Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0293531Z 2025-10-10T01:45:59.0294101Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:59.0294710Z 2025-10-10T01:45:59.0295014Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:45:59.0295315Z 2025-10-10T01:45:59.0295585Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0295862Z 2025-10-10T01:45:59.0296731Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:59.0297292Z 2025-10-10T01:45:59.0297583Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:45:59.0297864Z 2025-10-10T01:45:59.0298134Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0298399Z 2025-10-10T01:45:59.0298990Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:59.0299514Z 2025-10-10T01:45:59.0299836Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:59.0300146Z 2025-10-10T01:45:59.0300416Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0300794Z 2025-10-10T01:45:59.0301317Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:59.0301809Z 2025-10-10T01:45:59.0302176Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:45:59.0302470Z 2025-10-10T01:45:59.0302733Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0303004Z 2025-10-10T01:45:59.0303522Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:59.0304002Z 2025-10-10T01:45:59.0304315Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:45:59.0304619Z 2025-10-10T01:45:59.0304845Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:45:59.0305083Z 2025-10-10T01:45:59.0305583Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:59.0306108Z 2025-10-10T01:45:59.0306356Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] raise RuntimeError( 2025-10-10T01:45:59.0306621Z 2025-10-10T01:45:59.0307188Z (EngineCore_DP0 pid=5266) ERROR 10-10 01:45:59 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:59.0307900Z (EngineCore_DP0 pid=5266) Process EngineCore_DP0: 2025-10-10T01:45:59.0308329Z (EngineCore_DP0 pid=5266) Traceback (most recent call last): 2025-10-10T01:45:59.0308947Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:45:59.0309480Z (EngineCore_DP0 pid=5266) self.run() 2025-10-10T01:45:59.0310017Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:45:59.0310642Z (EngineCore_DP0 pid=5266) self._target(*self._args, **self._kwargs) 2025-10-10T01:45:59.0311318Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:45:59.0311866Z (EngineCore_DP0 pid=5266) raise e 2025-10-10T01:45:59.0312455Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:45:59.0313096Z (EngineCore_DP0 pid=5266) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:45:59.0313536Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0314156Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:45:59.0314802Z (EngineCore_DP0 pid=5266) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:45:59.0315476Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:45:59.0316097Z (EngineCore_DP0 pid=5266) self.model_executor = executor_class(vllm_config) 2025-10-10T01:45:59.0316545Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0317197Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:45:59.0317843Z (EngineCore_DP0 pid=5266) self._init_executor() 2025-10-10T01:45:59.0318549Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:45:59.0319351Z (EngineCore_DP0 pid=5266) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:45:59.0320073Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:45:59.0320782Z (EngineCore_DP0 pid=5266) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:45:59.0321274Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0321915Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:45:59.0322509Z (EngineCore_DP0 pid=5266) return func(*args, **kwargs) 2025-10-10T01:45:59.0322895Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0323596Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:45:59.0324230Z (EngineCore_DP0 pid=5266) worker_class = resolve_obj_by_qualname( 2025-10-10T01:45:59.0324653Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0325322Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:45:59.0325977Z (EngineCore_DP0 pid=5266) module = importlib.import_module(module_name) 2025-10-10T01:45:59.0326419Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0327002Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:45:59.0327632Z (EngineCore_DP0 pid=5266) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:45:59.0328171Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0328672Z (EngineCore_DP0 pid=5266) File "", line 1387, in _gcd_import 2025-10-10T01:45:59.0329218Z (EngineCore_DP0 pid=5266) File "", line 1360, in _find_and_load 2025-10-10T01:45:59.0329808Z (EngineCore_DP0 pid=5266) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:45:59.0330387Z (EngineCore_DP0 pid=5266) File "", line 935, in _load_unlocked 2025-10-10T01:45:59.0330955Z (EngineCore_DP0 pid=5266) File "", line 999, in exec_module 2025-10-10T01:45:59.0331551Z (EngineCore_DP0 pid=5266) File "", line 488, in _call_with_frames_removed 2025-10-10T01:45:59.0332282Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:45:59.0332958Z (EngineCore_DP0 pid=5266) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:45:59.0333672Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:45:59.0334387Z (EngineCore_DP0 pid=5266) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:45:59.0335215Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:45:59.0335895Z (EngineCore_DP0 pid=5266) class FlashAttentionMetadataBuilder( 2025-10-10T01:45:59.0336729Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:45:59.0337550Z (EngineCore_DP0 pid=5266) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:45:59.0338026Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0338711Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:45:59.0339389Z (EngineCore_DP0 pid=5266) if not is_fa_version_supported(fa_version): 2025-10-10T01:45:59.0339824Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0340569Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:45:59.0341310Z (EngineCore_DP0 pid=5266) return _is_fa2_supported(device)[0] 2025-10-10T01:45:59.0341734Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0342442Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:45:59.0343155Z (EngineCore_DP0 pid=5266) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:45:59.0343621Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0344290Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:45:59.0344927Z (EngineCore_DP0 pid=5266) prop = get_device_properties(device) 2025-10-10T01:45:59.0345348Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:45:59.0346050Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:45:59.0346719Z (EngineCore_DP0 pid=5266) _lazy_init() # will define _get_device_properties 2025-10-10T01:45:59.0347118Z (EngineCore_DP0 pid=5266) ^^^^^^^^^^^^ 2025-10-10T01:45:59.0347700Z (EngineCore_DP0 pid=5266) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:45:59.0348262Z (EngineCore_DP0 pid=5266) raise RuntimeError( 2025-10-10T01:45:59.0348947Z (EngineCore_DP0 pid=5266) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:45:59.4250384Z FAILED 2025-10-10T01:45:59.4380016Z models/test_initialization.py::test_can_initialize_large_subset[NomicBertModel] Fork a new process to run a test 5270 2025-10-10T01:45:59.4391010Z Fork a new process to run a test 0 2025-10-10T01:45:59.4662321Z INFO 10-10 01:45:59 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='NomicBertModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nomic-ai/nomic-embed-text-v2-moe'} 2025-10-10T01:45:59.5778851Z 2025-10-10T01:45:59.5781165Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:45:59.5781854Z config.json: 2.48kB [00:00, 11.5MB/s] 2025-10-10T01:45:59.6581156Z 2025-10-10T01:45:59.6582219Z configuration_hf_nomic_bert.py: 0.00B [00:00, ?B/s] 2025-10-10T01:45:59.6582880Z configuration_hf_nomic_bert.py: 1.96kB [00:00, 21.1MB/s] 2025-10-10T01:45:59.6672549Z A new version of the following files was downloaded from https://huggingface.co/nomic-ai/nomic-bert-2048: 2025-10-10T01:45:59.6673408Z - configuration_hf_nomic_bert.py 2025-10-10T01:45:59.6674046Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:45:59.8969342Z 2025-10-10T01:45:59.8969957Z sentence_bert_config.json: 0% 0.00/53.0 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:08.3841671Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:08.3842105Z 2025-10-10T01:46:08.3842788Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:08.3843204Z 2025-10-10T01:46:08.3843883Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:08.3844490Z 2025-10-10T01:46:08.3844857Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:08.3845175Z 2025-10-10T01:46:08.3845459Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3845741Z 2025-10-10T01:46:08.3846227Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:08.3846681Z 2025-10-10T01:46:08.3847025Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:08.3847352Z 2025-10-10T01:46:08.3847840Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:08.3848277Z 2025-10-10T01:46:08.3848596Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:08.3849007Z 2025-10-10T01:46:08.3849292Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3849566Z 2025-10-10T01:46:08.3850205Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:08.3850694Z 2025-10-10T01:46:08.3850955Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] self._init_executor() 2025-10-10T01:46:08.3851224Z 2025-10-10T01:46:08.3851791Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:08.3852310Z 2025-10-10T01:46:08.3852632Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:08.3852959Z 2025-10-10T01:46:08.3853661Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:08.3854505Z 2025-10-10T01:46:08.3855106Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:08.3855798Z 2025-10-10T01:46:08.3856355Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3856655Z 2025-10-10T01:46:08.3857322Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:08.3857786Z 2025-10-10T01:46:08.3858069Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:08.3858347Z 2025-10-10T01:46:08.3858601Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3858867Z 2025-10-10T01:46:08.3859385Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:08.3859933Z 2025-10-10T01:46:08.3860231Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:08.3860528Z 2025-10-10T01:46:08.3860791Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3861064Z 2025-10-10T01:46:08.3861585Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:08.3862083Z 2025-10-10T01:46:08.3862398Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:08.3862698Z 2025-10-10T01:46:08.3862978Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3863254Z 2025-10-10T01:46:08.3863704Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:08.3864113Z 2025-10-10T01:46:08.3864459Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:08.3864787Z 2025-10-10T01:46:08.3865069Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3865399Z 2025-10-10T01:46:08.3865741Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:08.3866089Z 2025-10-10T01:46:08.3866475Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:08.3866822Z 2025-10-10T01:46:08.3867198Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:08.3867555Z 2025-10-10T01:46:08.3867899Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:08.3868228Z 2025-10-10T01:46:08.3868590Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:08.3868940Z 2025-10-10T01:46:08.3869314Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:08.3869669Z 2025-10-10T01:46:08.3870167Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:08.3870678Z 2025-10-10T01:46:08.3871027Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:08.3871362Z 2025-10-10T01:46:08.3871874Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:08.3872353Z 2025-10-10T01:46:08.3872720Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:08.3873082Z 2025-10-10T01:46:08.3873654Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:08.3874237Z 2025-10-10T01:46:08.3874636Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:08.3874996Z 2025-10-10T01:46:08.3875739Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:08.3876369Z 2025-10-10T01:46:08.3876736Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:08.3877084Z 2025-10-10T01:46:08.3877341Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3877599Z 2025-10-10T01:46:08.3878160Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:08.3878681Z 2025-10-10T01:46:08.3879096Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:08.3879406Z 2025-10-10T01:46:08.3879679Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3879954Z 2025-10-10T01:46:08.3880547Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:08.3881147Z 2025-10-10T01:46:08.3881426Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:08.3881705Z 2025-10-10T01:46:08.3882005Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3882274Z 2025-10-10T01:46:08.3882857Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:08.3883377Z 2025-10-10T01:46:08.3883698Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:08.3884179Z 2025-10-10T01:46:08.3884500Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3884843Z 2025-10-10T01:46:08.3885467Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:08.3886003Z 2025-10-10T01:46:08.3886285Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:08.3886618Z 2025-10-10T01:46:08.3886880Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3887145Z 2025-10-10T01:46:08.3887674Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:08.3888149Z 2025-10-10T01:46:08.3888459Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:08.3888765Z 2025-10-10T01:46:08.3888994Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:08.3889232Z 2025-10-10T01:46:08.3889711Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:08.3890167Z 2025-10-10T01:46:08.3890452Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] raise RuntimeError( 2025-10-10T01:46:08.3890714Z 2025-10-10T01:46:08.3891280Z (EngineCore_DP0 pid=5352) ERROR 10-10 01:46:08 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:08.3891975Z (EngineCore_DP0 pid=5352) Process EngineCore_DP0: 2025-10-10T01:46:08.3892404Z (EngineCore_DP0 pid=5352) Traceback (most recent call last): 2025-10-10T01:46:08.3893015Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:08.3893530Z (EngineCore_DP0 pid=5352) self.run() 2025-10-10T01:46:08.3894059Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:08.3894631Z (EngineCore_DP0 pid=5352) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:08.3895302Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:08.3895844Z (EngineCore_DP0 pid=5352) raise e 2025-10-10T01:46:08.3896633Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:08.3897272Z (EngineCore_DP0 pid=5352) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:08.3897839Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3898524Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:08.3899179Z (EngineCore_DP0 pid=5352) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:08.3899843Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:08.3900459Z (EngineCore_DP0 pid=5352) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:08.3900902Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3901549Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:08.3902153Z (EngineCore_DP0 pid=5352) self._init_executor() 2025-10-10T01:46:08.3902819Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:08.3903570Z (EngineCore_DP0 pid=5352) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:08.3904300Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:08.3905020Z (EngineCore_DP0 pid=5352) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:08.3905525Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3906166Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:08.3906754Z (EngineCore_DP0 pid=5352) return func(*args, **kwargs) 2025-10-10T01:46:08.3907136Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3907773Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:08.3908471Z (EngineCore_DP0 pid=5352) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:08.3908901Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3909565Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:08.3910223Z (EngineCore_DP0 pid=5352) module = importlib.import_module(module_name) 2025-10-10T01:46:08.3910669Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3911250Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:08.3911874Z (EngineCore_DP0 pid=5352) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:08.3912368Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3912866Z (EngineCore_DP0 pid=5352) File "", line 1387, in _gcd_import 2025-10-10T01:46:08.3913414Z (EngineCore_DP0 pid=5352) File "", line 1360, in _find_and_load 2025-10-10T01:46:08.3913998Z (EngineCore_DP0 pid=5352) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:08.3914585Z (EngineCore_DP0 pid=5352) File "", line 935, in _load_unlocked 2025-10-10T01:46:08.3915241Z (EngineCore_DP0 pid=5352) File "", line 999, in exec_module 2025-10-10T01:46:08.3915842Z (EngineCore_DP0 pid=5352) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:08.3916613Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:08.3917294Z (EngineCore_DP0 pid=5352) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:08.3918027Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:08.3918745Z (EngineCore_DP0 pid=5352) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:08.3919632Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:08.3920304Z (EngineCore_DP0 pid=5352) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:08.3921092Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:08.3921995Z (EngineCore_DP0 pid=5352) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:08.3922475Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3923189Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:08.3923879Z (EngineCore_DP0 pid=5352) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:08.3924315Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3925065Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:08.3925767Z (EngineCore_DP0 pid=5352) return _is_fa2_supported(device)[0] 2025-10-10T01:46:08.3926230Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3926947Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:08.3927651Z (EngineCore_DP0 pid=5352) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:08.3928103Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3928776Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:08.3929410Z (EngineCore_DP0 pid=5352) prop = get_device_properties(device) 2025-10-10T01:46:08.3929824Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:08.3930480Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:08.3931136Z (EngineCore_DP0 pid=5352) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:08.3931526Z (EngineCore_DP0 pid=5352) ^^^^^^^^^^^^ 2025-10-10T01:46:08.3932115Z (EngineCore_DP0 pid=5352) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:08.3932678Z (EngineCore_DP0 pid=5352) raise RuntimeError( 2025-10-10T01:46:08.3933402Z (EngineCore_DP0 pid=5352) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:08.8180060Z FAILED 2025-10-10T01:46:08.8311200Z models/test_initialization.py::test_can_initialize_large_subset[FalconMambaForCausalLM] Fork a new process to run a test 5356 2025-10-10T01:46:08.8322470Z Fork a new process to run a test 0 2025-10-10T01:46:08.8594971Z INFO 10-10 01:46:08 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='FalconMambaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'tiiuae/falcon-mamba-7b-instruct'} 2025-10-10T01:46:08.9476937Z 2025-10-10T01:46:08.9477799Z config.json: 0% 0.00/870 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:09.8776187Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:09.8777037Z 2025-10-10T01:46:09.8777473Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:09.8778023Z 2025-10-10T01:46:09.8778873Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:09.8779574Z 2025-10-10T01:46:09.8780058Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:09.8780574Z 2025-10-10T01:46:09.8780990Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8781438Z 2025-10-10T01:46:09.8782168Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:09.8782909Z 2025-10-10T01:46:09.8783638Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:09.8784117Z 2025-10-10T01:46:09.8784901Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:09.8785633Z 2025-10-10T01:46:09.8786159Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:09.8786502Z 2025-10-10T01:46:09.8786843Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8787261Z 2025-10-10T01:46:09.8788137Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:09.8788848Z 2025-10-10T01:46:09.8789170Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] self._init_executor() 2025-10-10T01:46:09.8789459Z 2025-10-10T01:46:09.8790241Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:09.8790916Z 2025-10-10T01:46:09.8791377Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:09.8791875Z 2025-10-10T01:46:09.8792438Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:09.8792946Z 2025-10-10T01:46:09.8793396Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:09.8793747Z 2025-10-10T01:46:09.8794064Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8794354Z 2025-10-10T01:46:09.8794859Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:09.8795314Z 2025-10-10T01:46:09.8795579Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:09.8795879Z 2025-10-10T01:46:09.8796350Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8796617Z 2025-10-10T01:46:09.8797140Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:09.8797720Z 2025-10-10T01:46:09.8798016Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:09.8798309Z 2025-10-10T01:46:09.8798575Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8798851Z 2025-10-10T01:46:09.8799506Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:09.8800002Z 2025-10-10T01:46:09.8800317Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:09.8800615Z 2025-10-10T01:46:09.8800899Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8801191Z 2025-10-10T01:46:09.8801703Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:09.8802167Z 2025-10-10T01:46:09.8802529Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:09.8802885Z 2025-10-10T01:46:09.8803176Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8803469Z 2025-10-10T01:46:09.8803810Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:09.8804137Z 2025-10-10T01:46:09.8804492Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:09.8804827Z 2025-10-10T01:46:09.8805209Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:09.8805577Z 2025-10-10T01:46:09.8805931Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:09.8806263Z 2025-10-10T01:46:09.8806637Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:09.8807074Z 2025-10-10T01:46:09.8807460Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:09.8807824Z 2025-10-10T01:46:09.8808390Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:09.8808876Z 2025-10-10T01:46:09.8809237Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:09.8809589Z 2025-10-10T01:46:09.8810122Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:09.8810606Z 2025-10-10T01:46:09.8810975Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:09.8811333Z 2025-10-10T01:46:09.8811887Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:09.8812449Z 2025-10-10T01:46:09.8812751Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:09.8813045Z 2025-10-10T01:46:09.8813672Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:09.8814237Z 2025-10-10T01:46:09.8814609Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:09.8814959Z 2025-10-10T01:46:09.8815226Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8815491Z 2025-10-10T01:46:09.8816059Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:09.8816578Z 2025-10-10T01:46:09.8816918Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:09.8817227Z 2025-10-10T01:46:09.8817499Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8817770Z 2025-10-10T01:46:09.8818369Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:09.8818920Z 2025-10-10T01:46:09.8819208Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:09.8819489Z 2025-10-10T01:46:09.8819750Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8820016Z 2025-10-10T01:46:09.8820595Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:09.8821129Z 2025-10-10T01:46:09.8821448Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:09.8821769Z 2025-10-10T01:46:09.8822044Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8822370Z 2025-10-10T01:46:09.8822890Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:09.8823374Z 2025-10-10T01:46:09.8823697Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:09.8823987Z 2025-10-10T01:46:09.8824258Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8824522Z 2025-10-10T01:46:09.8825043Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:09.8825578Z 2025-10-10T01:46:09.8825957Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:09.8826314Z 2025-10-10T01:46:09.8826584Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:09.8826867Z 2025-10-10T01:46:09.8827393Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:09.8827892Z 2025-10-10T01:46:09.8828142Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] raise RuntimeError( 2025-10-10T01:46:09.8828411Z 2025-10-10T01:46:09.8828973Z (EngineCore_DP0 pid=5364) ERROR 10-10 01:46:09 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:09.8829663Z (EngineCore_DP0 pid=5364) Process EngineCore_DP0: 2025-10-10T01:46:09.8830079Z (EngineCore_DP0 pid=5364) Traceback (most recent call last): 2025-10-10T01:46:09.8830685Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:09.8831200Z (EngineCore_DP0 pid=5364) self.run() 2025-10-10T01:46:09.8831729Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:09.8832346Z (EngineCore_DP0 pid=5364) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:09.8833027Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:09.8833586Z (EngineCore_DP0 pid=5364) raise e 2025-10-10T01:46:09.8834183Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:09.8834818Z (EngineCore_DP0 pid=5364) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:09.8835275Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8835896Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:09.8836540Z (EngineCore_DP0 pid=5364) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:09.8837201Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:09.8837810Z (EngineCore_DP0 pid=5364) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:09.8838262Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8838920Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:09.8839655Z (EngineCore_DP0 pid=5364) self._init_executor() 2025-10-10T01:46:09.8840327Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:09.8841066Z (EngineCore_DP0 pid=5364) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:09.8841793Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:09.8842511Z (EngineCore_DP0 pid=5364) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:09.8843006Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8843642Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:09.8844225Z (EngineCore_DP0 pid=5364) return func(*args, **kwargs) 2025-10-10T01:46:09.8844603Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8845242Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:09.8845930Z (EngineCore_DP0 pid=5364) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:09.8846362Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8847031Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:09.8847678Z (EngineCore_DP0 pid=5364) module = importlib.import_module(module_name) 2025-10-10T01:46:09.8848124Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8848701Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:09.8849327Z (EngineCore_DP0 pid=5364) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:09.8849870Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8850381Z (EngineCore_DP0 pid=5364) File "", line 1387, in _gcd_import 2025-10-10T01:46:09.8850936Z (EngineCore_DP0 pid=5364) File "", line 1360, in _find_and_load 2025-10-10T01:46:09.8851520Z (EngineCore_DP0 pid=5364) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:09.8852093Z (EngineCore_DP0 pid=5364) File "", line 935, in _load_unlocked 2025-10-10T01:46:09.8852670Z (EngineCore_DP0 pid=5364) File "", line 999, in exec_module 2025-10-10T01:46:09.8853270Z (EngineCore_DP0 pid=5364) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:09.8854004Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:09.8854678Z (EngineCore_DP0 pid=5364) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:09.8855393Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:09.8856111Z (EngineCore_DP0 pid=5364) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:09.8856889Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:09.8857608Z (EngineCore_DP0 pid=5364) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:09.8858466Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:09.8859303Z (EngineCore_DP0 pid=5364) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:09.8859781Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8860468Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:09.8861150Z (EngineCore_DP0 pid=5364) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:09.8861593Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8862340Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:09.8863091Z (EngineCore_DP0 pid=5364) return _is_fa2_supported(device)[0] 2025-10-10T01:46:09.8863506Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8864215Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:09.8872359Z (EngineCore_DP0 pid=5364) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:09.8872969Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8873702Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:09.8874400Z (EngineCore_DP0 pid=5364) prop = get_device_properties(device) 2025-10-10T01:46:09.8874847Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:09.8875618Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:09.8876315Z (EngineCore_DP0 pid=5364) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:09.8876737Z (EngineCore_DP0 pid=5364) ^^^^^^^^^^^^ 2025-10-10T01:46:09.8877326Z (EngineCore_DP0 pid=5364) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:09.8877889Z (EngineCore_DP0 pid=5364) raise RuntimeError( 2025-10-10T01:46:09.8878583Z (EngineCore_DP0 pid=5364) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:10.2851806Z FAILED 2025-10-10T01:46:10.2981666Z models/test_initialization.py::test_can_initialize_large_subset[AyaVisionForConditionalGeneration] Fork a new process to run a test 5368 2025-10-10T01:46:10.2993661Z Fork a new process to run a test 0 2025-10-10T01:46:10.3268464Z INFO 10-10 01:46:10 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='AyaVisionForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'CohereForAI/aya-vision-8b'} 2025-10-10T01:46:10.5174764Z 2025-10-10T01:46:10.5175909Z config.json: 0% 0.00/1.17k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:20.2441155Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:20.2442074Z 2025-10-10T01:46:20.2442613Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:20.2443131Z 2025-10-10T01:46:20.2444224Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:20.2445114Z 2025-10-10T01:46:20.2445702Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:20.2446272Z 2025-10-10T01:46:20.2446801Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2447341Z 2025-10-10T01:46:20.2448292Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:20.2449214Z 2025-10-10T01:46:20.2449899Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:20.2450533Z 2025-10-10T01:46:20.2451500Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:20.2452553Z 2025-10-10T01:46:20.2453222Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:20.2453817Z 2025-10-10T01:46:20.2454383Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2454897Z 2025-10-10T01:46:20.2455886Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:20.2456767Z 2025-10-10T01:46:20.2457239Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] self._init_executor() 2025-10-10T01:46:20.2457715Z 2025-10-10T01:46:20.2458574Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:20.2459379Z 2025-10-10T01:46:20.2460125Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:20.2460701Z 2025-10-10T01:46:20.2461505Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:20.2462027Z 2025-10-10T01:46:20.2462394Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:20.2462751Z 2025-10-10T01:46:20.2463054Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2463334Z 2025-10-10T01:46:20.2463850Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:20.2464315Z 2025-10-10T01:46:20.2464589Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:20.2464858Z 2025-10-10T01:46:20.2465114Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2465370Z 2025-10-10T01:46:20.2465888Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:20.2466462Z 2025-10-10T01:46:20.2466757Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:20.2467049Z 2025-10-10T01:46:20.2467358Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2467639Z 2025-10-10T01:46:20.2468177Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:20.2468664Z 2025-10-10T01:46:20.2468975Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:20.2469280Z 2025-10-10T01:46:20.2469560Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2469835Z 2025-10-10T01:46:20.2470283Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:20.2470689Z 2025-10-10T01:46:20.2471028Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:20.2471412Z 2025-10-10T01:46:20.2471703Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2472004Z 2025-10-10T01:46:20.2472354Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:20.2472693Z 2025-10-10T01:46:20.2473051Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:20.2473399Z 2025-10-10T01:46:20.2473774Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:20.2474132Z 2025-10-10T01:46:20.2474495Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:20.2474830Z 2025-10-10T01:46:20.2475241Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:20.2475595Z 2025-10-10T01:46:20.2475975Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:20.2476335Z 2025-10-10T01:46:20.2476839Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:20.2477312Z 2025-10-10T01:46:20.2477663Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:20.2478010Z 2025-10-10T01:46:20.2478550Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:20.2479173Z 2025-10-10T01:46:20.2479558Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:20.2479927Z 2025-10-10T01:46:20.2480491Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:20.2481025Z 2025-10-10T01:46:20.2481377Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:20.2481675Z 2025-10-10T01:46:20.2482368Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:20.2482946Z 2025-10-10T01:46:20.2483323Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:20.2483676Z 2025-10-10T01:46:20.2483937Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2484201Z 2025-10-10T01:46:20.2484757Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:20.2485288Z 2025-10-10T01:46:20.2485580Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:20.2485880Z 2025-10-10T01:46:20.2486163Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2486484Z 2025-10-10T01:46:20.2487088Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:20.2487639Z 2025-10-10T01:46:20.2487917Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:20.2488217Z 2025-10-10T01:46:20.2488486Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2488756Z 2025-10-10T01:46:20.2489354Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:20.2489890Z 2025-10-10T01:46:20.2490218Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:20.2490535Z 2025-10-10T01:46:20.2490868Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2491153Z 2025-10-10T01:46:20.2491677Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:20.2492165Z 2025-10-10T01:46:20.2492444Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:20.2492734Z 2025-10-10T01:46:20.2492994Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2493268Z 2025-10-10T01:46:20.2493785Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:20.2494298Z 2025-10-10T01:46:20.2494611Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:20.2494920Z 2025-10-10T01:46:20.2495142Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:20.2495382Z 2025-10-10T01:46:20.2495872Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:20.2496656Z 2025-10-10T01:46:20.2496938Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] raise RuntimeError( 2025-10-10T01:46:20.2497204Z 2025-10-10T01:46:20.2497877Z (EngineCore_DP0 pid=5446) ERROR 10-10 01:46:20 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:20.2498590Z (EngineCore_DP0 pid=5446) Process EngineCore_DP0: 2025-10-10T01:46:20.2499013Z (EngineCore_DP0 pid=5446) Traceback (most recent call last): 2025-10-10T01:46:20.2499622Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:20.2500143Z (EngineCore_DP0 pid=5446) self.run() 2025-10-10T01:46:20.2500682Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:20.2501276Z (EngineCore_DP0 pid=5446) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:20.2501948Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:20.2502571Z (EngineCore_DP0 pid=5446) raise e 2025-10-10T01:46:20.2503162Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:20.2503793Z (EngineCore_DP0 pid=5446) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:20.2504232Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2504852Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:20.2505494Z (EngineCore_DP0 pid=5446) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:20.2506152Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:20.2506771Z (EngineCore_DP0 pid=5446) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:20.2507291Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2507956Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:20.2508532Z (EngineCore_DP0 pid=5446) self._init_executor() 2025-10-10T01:46:20.2509197Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:20.2509883Z (EngineCore_DP0 pid=5446) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:20.2510624Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:20.2511341Z (EngineCore_DP0 pid=5446) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:20.2511835Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2512480Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:20.2513067Z (EngineCore_DP0 pid=5446) return func(*args, **kwargs) 2025-10-10T01:46:20.2513453Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2514087Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:20.2514785Z (EngineCore_DP0 pid=5446) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:20.2515212Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2515923Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:20.2516584Z (EngineCore_DP0 pid=5446) module = importlib.import_module(module_name) 2025-10-10T01:46:20.2517033Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2517607Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:20.2518236Z (EngineCore_DP0 pid=5446) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:20.2518735Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2519333Z (EngineCore_DP0 pid=5446) File "", line 1387, in _gcd_import 2025-10-10T01:46:20.2519906Z (EngineCore_DP0 pid=5446) File "", line 1360, in _find_and_load 2025-10-10T01:46:20.2520554Z (EngineCore_DP0 pid=5446) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:20.2521135Z (EngineCore_DP0 pid=5446) File "", line 935, in _load_unlocked 2025-10-10T01:46:20.2521709Z (EngineCore_DP0 pid=5446) File "", line 999, in exec_module 2025-10-10T01:46:20.2522315Z (EngineCore_DP0 pid=5446) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:20.2523052Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:20.2523737Z (EngineCore_DP0 pid=5446) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:20.2524459Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:20.2525232Z (EngineCore_DP0 pid=5446) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:20.2526022Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:20.2526697Z (EngineCore_DP0 pid=5446) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:20.2527487Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:20.2528306Z (EngineCore_DP0 pid=5446) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:20.2528791Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2529484Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:20.2530166Z (EngineCore_DP0 pid=5446) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:20.2530605Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2531342Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:20.2532036Z (EngineCore_DP0 pid=5446) return _is_fa2_supported(device)[0] 2025-10-10T01:46:20.2532504Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2533252Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:20.2533971Z (EngineCore_DP0 pid=5446) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:20.2534424Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2535086Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:20.2535727Z (EngineCore_DP0 pid=5446) prop = get_device_properties(device) 2025-10-10T01:46:20.2536141Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:20.2536796Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:20.2537456Z (EngineCore_DP0 pid=5446) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:20.2537892Z (EngineCore_DP0 pid=5446) ^^^^^^^^^^^^ 2025-10-10T01:46:20.2538471Z (EngineCore_DP0 pid=5446) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:20.2539039Z (EngineCore_DP0 pid=5446) raise RuntimeError( 2025-10-10T01:46:20.2539723Z (EngineCore_DP0 pid=5446) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:20.6828738Z FAILED 2025-10-10T01:46:20.6959362Z models/test_initialization.py::test_can_initialize_large_subset[Qwen3NextMTP] Fork a new process to run a test 5450 2025-10-10T01:46:20.6971055Z Fork a new process to run a test 0 2025-10-10T01:46:20.6974642Z `transformers==4.56.2` installed, but `transformers>=4.56.3` is required to run this model. 2025-10-10T01:46:20.9974699Z PASSED 2025-10-10T01:46:21.0104360Z models/test_initialization.py::test_can_initialize_large_subset[FalconH1ForCausalLM] Fork a new process to run a test 5451 2025-10-10T01:46:21.0115217Z Fork a new process to run a test 0 2025-10-10T01:46:21.0387757Z INFO 10-10 01:46:21 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='FalconH1ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'tiiuae/Falcon-H1-0.5B-Base'} 2025-10-10T01:46:21.1139098Z 2025-10-10T01:46:21.1140995Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:21.1141381Z config.json: 1.58kB [00:00, 8.98MB/s] 2025-10-10T01:46:27.8338686Z INFO 10-10 01:46:27 [model.py:551] Resolved architecture: FalconH1ForCausalLM 2025-10-10T01:46:27.8339189Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:46:27.8584345Z INFO 10-10 01:46:27 [model.py:1545] Using max model len 16384 2025-10-10T01:46:28.0310714Z INFO 10-10 01:46:28 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:46:28.0312316Z INFO 10-10 01:46:28 [config.py:297] Hybrid or mamba-based model detected: disabling prefix caching since it is not yet supported. 2025-10-10T01:46:28.0313080Z INFO 10-10 01:46:28 [config.py:308] Hybrid or mamba-based model detected: setting cudagraph mode to FULL_AND_PIECEWISE in order to optimize performance. 2025-10-10T01:46:28.0843563Z INFO 10-10 01:46:28 [config.py:376] Setting attention block size to 800 tokens to ensure that attention page size is >= mamba page size. 2025-10-10T01:46:28.0845989Z INFO 10-10 01:46:28 [config.py:397] Padding mamba page size by 1.39% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:46:28.1217122Z 2025-10-10T01:46:28.1221897Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:28.1222353Z tokenizer_config.json: 99.7kB [00:00, 221MB/s] 2025-10-10T01:46:28.1875860Z 2025-10-10T01:46:28.1995995Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:28.1996682Z tokenizer.json: 2.35MB [00:00, 196MB/s] 2025-10-10T01:46:28.2966530Z 2025-10-10T01:46:28.2968209Z special_tokens_map.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:28.2968630Z special_tokens_map.json: 7.42kB [00:00, 53.3MB/s] 2025-10-10T01:46:28.4615868Z 2025-10-10T01:46:28.4616344Z generation_config.json: 0% 0.00/138 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:28.5879429Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:28.5880032Z 2025-10-10T01:46:28.5880602Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:28.5881139Z 2025-10-10T01:46:28.5882101Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:28.5882974Z 2025-10-10T01:46:28.5883547Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:28.5884093Z 2025-10-10T01:46:28.5884599Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5885347Z 2025-10-10T01:46:28.5886271Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:28.5887120Z 2025-10-10T01:46:28.5887944Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:28.5888618Z 2025-10-10T01:46:28.5889619Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:28.5890506Z 2025-10-10T01:46:28.5891148Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:28.5891747Z 2025-10-10T01:46:28.5892312Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5892888Z 2025-10-10T01:46:28.5893883Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:28.5894760Z 2025-10-10T01:46:28.5895459Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] self._init_executor() 2025-10-10T01:46:28.5895947Z 2025-10-10T01:46:28.5897270Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:28.5898220Z 2025-10-10T01:46:28.5898812Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:28.5899358Z 2025-10-10T01:46:28.5900345Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:28.5901202Z 2025-10-10T01:46:28.5901732Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:28.5902102Z 2025-10-10T01:46:28.5902557Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5902889Z 2025-10-10T01:46:28.5903432Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:28.5903918Z 2025-10-10T01:46:28.5904199Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:28.5904475Z 2025-10-10T01:46:28.5904758Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5905022Z 2025-10-10T01:46:28.5905573Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:28.5906055Z 2025-10-10T01:46:28.5906369Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:28.5906666Z 2025-10-10T01:46:28.5906950Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5907240Z 2025-10-10T01:46:28.5907772Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:28.5908262Z 2025-10-10T01:46:28.5908578Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:28.5909000Z 2025-10-10T01:46:28.5909292Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5909571Z 2025-10-10T01:46:28.5910090Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:28.5910519Z 2025-10-10T01:46:28.5910890Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:28.5911219Z 2025-10-10T01:46:28.5911519Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5911793Z 2025-10-10T01:46:28.5912149Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:28.5912479Z 2025-10-10T01:46:28.5912834Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:28.5913172Z 2025-10-10T01:46:28.5913569Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:28.5914022Z 2025-10-10T01:46:28.5914382Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:28.5914722Z 2025-10-10T01:46:28.5915096Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:28.5915462Z 2025-10-10T01:46:28.5915838Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:28.5916195Z 2025-10-10T01:46:28.5916709Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:28.5917170Z 2025-10-10T01:46:28.5917587Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:28.5917926Z 2025-10-10T01:46:28.5918459Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:28.5918934Z 2025-10-10T01:46:28.5919430Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:28.5919803Z 2025-10-10T01:46:28.5920364Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:28.5920872Z 2025-10-10T01:46:28.5921188Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:28.5921511Z 2025-10-10T01:46:28.5922164Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:28.5922745Z 2025-10-10T01:46:28.5923124Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:28.5923479Z 2025-10-10T01:46:28.5923743Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5924068Z 2025-10-10T01:46:28.5924646Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:28.5925161Z 2025-10-10T01:46:28.5925513Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:28.5925820Z 2025-10-10T01:46:28.5926109Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5926382Z 2025-10-10T01:46:28.5927003Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:28.5927559Z 2025-10-10T01:46:28.5927852Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:28.5928143Z 2025-10-10T01:46:28.5928412Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5928681Z 2025-10-10T01:46:28.5929271Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:28.5929859Z 2025-10-10T01:46:28.5930190Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:28.5930502Z 2025-10-10T01:46:28.5930791Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5931063Z 2025-10-10T01:46:28.5931601Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:28.5932083Z 2025-10-10T01:46:28.5932373Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:28.5932663Z 2025-10-10T01:46:28.5932946Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5933227Z 2025-10-10T01:46:28.5933812Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:28.5934347Z 2025-10-10T01:46:28.5934675Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:28.5934984Z 2025-10-10T01:46:28.5935220Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:28.5935461Z 2025-10-10T01:46:28.5935967Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:28.5936415Z 2025-10-10T01:46:28.5936690Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] raise RuntimeError( 2025-10-10T01:46:28.5936954Z 2025-10-10T01:46:28.5937543Z (EngineCore_DP0 pid=5509) ERROR 10-10 01:46:28 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:28.5938276Z (EngineCore_DP0 pid=5509) Process EngineCore_DP0: 2025-10-10T01:46:28.5938687Z (EngineCore_DP0 pid=5509) Traceback (most recent call last): 2025-10-10T01:46:28.5939303Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:28.5939897Z (EngineCore_DP0 pid=5509) self.run() 2025-10-10T01:46:28.5940444Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:28.5941066Z (EngineCore_DP0 pid=5509) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:28.5941773Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:28.5942329Z (EngineCore_DP0 pid=5509) raise e 2025-10-10T01:46:28.5942936Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:28.5943588Z (EngineCore_DP0 pid=5509) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:28.5944043Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5944678Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:28.5945320Z (EngineCore_DP0 pid=5509) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:28.5946081Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:28.5946712Z (EngineCore_DP0 pid=5509) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:28.5947176Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5947849Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:28.5948441Z (EngineCore_DP0 pid=5509) self._init_executor() 2025-10-10T01:46:28.5949117Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:28.5949815Z (EngineCore_DP0 pid=5509) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:28.5950593Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:28.5951320Z (EngineCore_DP0 pid=5509) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:28.5951831Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5952473Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:28.5953074Z (EngineCore_DP0 pid=5509) return func(*args, **kwargs) 2025-10-10T01:46:28.5953474Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5954127Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:28.5954789Z (EngineCore_DP0 pid=5509) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:28.5955221Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5955901Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:28.5956569Z (EngineCore_DP0 pid=5509) module = importlib.import_module(module_name) 2025-10-10T01:46:28.5957027Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5957618Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:28.5958307Z (EngineCore_DP0 pid=5509) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:28.5958869Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5959464Z (EngineCore_DP0 pid=5509) File "", line 1387, in _gcd_import 2025-10-10T01:46:28.5960034Z (EngineCore_DP0 pid=5509) File "", line 1360, in _find_and_load 2025-10-10T01:46:28.5960633Z (EngineCore_DP0 pid=5509) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:28.5961224Z (EngineCore_DP0 pid=5509) File "", line 935, in _load_unlocked 2025-10-10T01:46:28.5961809Z (EngineCore_DP0 pid=5509) File "", line 999, in exec_module 2025-10-10T01:46:28.5962426Z (EngineCore_DP0 pid=5509) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:28.5963180Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:28.5963936Z (EngineCore_DP0 pid=5509) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:28.5964664Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:28.5965398Z (EngineCore_DP0 pid=5509) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:28.5966198Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:28.5966884Z (EngineCore_DP0 pid=5509) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:28.5967693Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:28.5968523Z (EngineCore_DP0 pid=5509) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:28.5969058Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5969771Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:28.5970472Z (EngineCore_DP0 pid=5509) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:28.5970922Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5971687Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:28.5972391Z (EngineCore_DP0 pid=5509) return _is_fa2_supported(device)[0] 2025-10-10T01:46:28.5972821Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5973559Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:28.5974287Z (EngineCore_DP0 pid=5509) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:28.5974753Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5975425Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:28.5976147Z (EngineCore_DP0 pid=5509) prop = get_device_properties(device) 2025-10-10T01:46:28.5976578Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:28.5977286Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:28.5977976Z (EngineCore_DP0 pid=5509) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:28.5978384Z (EngineCore_DP0 pid=5509) ^^^^^^^^^^^^ 2025-10-10T01:46:28.5978984Z (EngineCore_DP0 pid=5509) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:28.5979558Z (EngineCore_DP0 pid=5509) raise RuntimeError( 2025-10-10T01:46:28.5980253Z (EngineCore_DP0 pid=5509) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:28.9898126Z FAILED 2025-10-10T01:46:29.0026765Z models/test_initialization.py::test_can_initialize_large_subset[MPTForCausalLM] Fork a new process to run a test 5513 2025-10-10T01:46:29.0038177Z Fork a new process to run a test 0 2025-10-10T01:46:29.0318586Z INFO 10-10 01:46:29 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MPTForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mosaicml/mpt-7b'} 2025-10-10T01:46:29.1675167Z 2025-10-10T01:46:29.1677512Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:29.1677839Z config.json: 1.23kB [00:00, 6.18MB/s] 2025-10-10T01:46:35.8788431Z INFO 10-10 01:46:35 [model.py:551] Resolved architecture: MPTForCausalLM 2025-10-10T01:46:35.8788919Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:46:35.9030933Z INFO 10-10 01:46:35 [model.py:1545] Using max model len 2048 2025-10-10T01:46:36.0718707Z INFO 10-10 01:46:36 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:46:36.1132976Z 2025-10-10T01:46:36.1133938Z tokenizer_config.json: 0% 0.00/237 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:36.6714046Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:36.6714932Z 2025-10-10T01:46:36.6715350Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:36.6715729Z 2025-10-10T01:46:36.6716404Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:36.6717003Z 2025-10-10T01:46:36.6717385Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:36.6717766Z 2025-10-10T01:46:36.6718119Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6718468Z 2025-10-10T01:46:36.6719199Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:36.6719772Z 2025-10-10T01:46:36.6720365Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:36.6720886Z 2025-10-10T01:46:36.6721415Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:36.6721865Z 2025-10-10T01:46:36.6722179Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:36.6722506Z 2025-10-10T01:46:36.6722784Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6723062Z 2025-10-10T01:46:36.6723570Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:36.6724147Z 2025-10-10T01:46:36.6724557Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] self._init_executor() 2025-10-10T01:46:36.6725051Z 2025-10-10T01:46:36.6726015Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:36.6726883Z 2025-10-10T01:46:36.6727454Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:36.6727951Z 2025-10-10T01:46:36.6728692Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:36.6729224Z 2025-10-10T01:46:36.6729712Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:36.6730142Z 2025-10-10T01:46:36.6730506Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6730861Z 2025-10-10T01:46:36.6731477Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:36.6732031Z 2025-10-10T01:46:36.6732312Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:36.6732596Z 2025-10-10T01:46:36.6732844Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6733229Z 2025-10-10T01:46:36.6733775Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:36.6734498Z 2025-10-10T01:46:36.6734829Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:36.6735124Z 2025-10-10T01:46:36.6735403Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6735679Z 2025-10-10T01:46:36.6736212Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:36.6736702Z 2025-10-10T01:46:36.6737004Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:36.6737313Z 2025-10-10T01:46:36.6737586Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6737866Z 2025-10-10T01:46:36.6738378Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:36.6738803Z 2025-10-10T01:46:36.6739146Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:36.6739484Z 2025-10-10T01:46:36.6739804Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6740140Z 2025-10-10T01:46:36.6740558Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:36.6740947Z 2025-10-10T01:46:36.6741364Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:36.6741734Z 2025-10-10T01:46:36.6742112Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:36.6742469Z 2025-10-10T01:46:36.6742811Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:36.6743145Z 2025-10-10T01:46:36.6743502Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:36.6743910Z 2025-10-10T01:46:36.6744285Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:36.6744649Z 2025-10-10T01:46:36.6745191Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:36.6745663Z 2025-10-10T01:46:36.6746024Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:36.6746363Z 2025-10-10T01:46:36.6746903Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:36.6747380Z 2025-10-10T01:46:36.6747758Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:36.6748117Z 2025-10-10T01:46:36.6748672Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:36.6749244Z 2025-10-10T01:46:36.6749542Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:36.6749845Z 2025-10-10T01:46:36.6750491Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:36.6751083Z 2025-10-10T01:46:36.6751449Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:36.6751808Z 2025-10-10T01:46:36.6752063Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6752337Z 2025-10-10T01:46:36.6752900Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:36.6753468Z 2025-10-10T01:46:36.6753779Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:36.6754085Z 2025-10-10T01:46:36.6754363Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6754634Z 2025-10-10T01:46:36.6755238Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:36.6755789Z 2025-10-10T01:46:36.6756073Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:36.6756356Z 2025-10-10T01:46:36.6756615Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6756887Z 2025-10-10T01:46:36.6757462Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:36.6757991Z 2025-10-10T01:46:36.6758306Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:36.6758629Z 2025-10-10T01:46:36.6758905Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6759358Z 2025-10-10T01:46:36.6759901Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:36.6760386Z 2025-10-10T01:46:36.6760727Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:36.6761029Z 2025-10-10T01:46:36.6761301Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6761580Z 2025-10-10T01:46:36.6762109Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:36.6762600Z 2025-10-10T01:46:36.6762911Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:36.6763222Z 2025-10-10T01:46:36.6763446Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:36.6763691Z 2025-10-10T01:46:36.6764171Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:36.6764698Z 2025-10-10T01:46:36.6764961Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] raise RuntimeError( 2025-10-10T01:46:36.6765230Z 2025-10-10T01:46:36.6765803Z (EngineCore_DP0 pid=5571) ERROR 10-10 01:46:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:36.6766519Z (EngineCore_DP0 pid=5571) Process EngineCore_DP0: 2025-10-10T01:46:36.6766927Z (EngineCore_DP0 pid=5571) Traceback (most recent call last): 2025-10-10T01:46:36.6767536Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:36.6768046Z (EngineCore_DP0 pid=5571) self.run() 2025-10-10T01:46:36.6768579Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:36.6769201Z (EngineCore_DP0 pid=5571) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:36.6769888Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:36.6770446Z (EngineCore_DP0 pid=5571) raise e 2025-10-10T01:46:36.6771030Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:36.6771680Z (EngineCore_DP0 pid=5571) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:36.6772121Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6772748Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:36.6773390Z (EngineCore_DP0 pid=5571) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:36.6774039Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:36.6774658Z (EngineCore_DP0 pid=5571) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:36.6775124Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6775774Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:36.6776428Z (EngineCore_DP0 pid=5571) self._init_executor() 2025-10-10T01:46:36.6777131Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:36.6777832Z (EngineCore_DP0 pid=5571) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:36.6778569Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:36.6779283Z (EngineCore_DP0 pid=5571) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:36.6779777Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6780419Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:36.6781011Z (EngineCore_DP0 pid=5571) return func(*args, **kwargs) 2025-10-10T01:46:36.6781409Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6782119Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:36.6782761Z (EngineCore_DP0 pid=5571) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:36.6783186Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6783850Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:36.6784507Z (EngineCore_DP0 pid=5571) module = importlib.import_module(module_name) 2025-10-10T01:46:36.6784953Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6785547Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:36.6786178Z (EngineCore_DP0 pid=5571) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:36.6786709Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6787220Z (EngineCore_DP0 pid=5571) File "", line 1387, in _gcd_import 2025-10-10T01:46:36.6787782Z (EngineCore_DP0 pid=5571) File "", line 1360, in _find_and_load 2025-10-10T01:46:36.6788370Z (EngineCore_DP0 pid=5571) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:36.6788950Z (EngineCore_DP0 pid=5571) File "", line 935, in _load_unlocked 2025-10-10T01:46:36.6789510Z (EngineCore_DP0 pid=5571) File "", line 999, in exec_module 2025-10-10T01:46:36.6790112Z (EngineCore_DP0 pid=5571) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:36.6790851Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:36.6791541Z (EngineCore_DP0 pid=5571) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:36.6792261Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:36.6792975Z (EngineCore_DP0 pid=5571) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:36.6793822Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:36.6794505Z (EngineCore_DP0 pid=5571) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:36.6795343Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:36.6796407Z (EngineCore_DP0 pid=5571) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:36.6796915Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6797605Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:36.6798293Z (EngineCore_DP0 pid=5571) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:36.6798739Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6799552Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:36.6800373Z (EngineCore_DP0 pid=5571) return _is_fa2_supported(device)[0] 2025-10-10T01:46:36.6800793Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6801523Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:36.6802240Z (EngineCore_DP0 pid=5571) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:36.6802689Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6803359Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:36.6803993Z (EngineCore_DP0 pid=5571) prop = get_device_properties(device) 2025-10-10T01:46:36.6804419Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:36.6805150Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:36.6805851Z (EngineCore_DP0 pid=5571) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:36.6806256Z (EngineCore_DP0 pid=5571) ^^^^^^^^^^^^ 2025-10-10T01:46:36.6806845Z (EngineCore_DP0 pid=5571) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:36.6807419Z (EngineCore_DP0 pid=5571) raise RuntimeError( 2025-10-10T01:46:36.6808100Z (EngineCore_DP0 pid=5571) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:37.0788245Z FAILED 2025-10-10T01:46:37.0916760Z models/test_initialization.py::test_can_initialize_large_subset[NVLM_D] Fork a new process to run a test 5575 2025-10-10T01:46:37.0927928Z Fork a new process to run a test 0 2025-10-10T01:46:37.1205700Z INFO 10-10 01:46:37 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='NVLM_D', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nvidia/NVLM-D-72B'} 2025-10-10T01:46:37.3617194Z 2025-10-10T01:46:37.3619332Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:37.3619968Z config.json: 3.71kB [00:00, 19.1MB/s] 2025-10-10T01:46:37.4344590Z 2025-10-10T01:46:37.4346282Z configuration_nvlm_d.py: 0.00B [00:00, ?B/s] 2025-10-10T01:46:37.4346723Z configuration_nvlm_d.py: 3.80kB [00:00, 34.2MB/s] 2025-10-10T01:46:37.4755749Z 2025-10-10T01:46:37.4757673Z configuration_intern_vit.py: 0.00B [00:00, ?B/s] 2025-10-10T01:46:37.4758353Z configuration_intern_vit.py: 5.55kB [00:00, 39.0MB/s] 2025-10-10T01:46:37.4779155Z A new version of the following files was downloaded from https://huggingface.co/nvidia/NVLM-D-72B: 2025-10-10T01:46:37.4779724Z - configuration_intern_vit.py 2025-10-10T01:46:37.4780282Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:46:37.4781368Z A new version of the following files was downloaded from https://huggingface.co/nvidia/NVLM-D-72B: 2025-10-10T01:46:37.4781961Z - configuration_nvlm_d.py 2025-10-10T01:46:37.4782249Z - configuration_intern_vit.py 2025-10-10T01:46:37.4782859Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:46:37.6996429Z 2025-10-10T01:46:37.6996912Z preprocessor_config.json: 0% 0.00/287 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:45.7531052Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:45.7531602Z 2025-10-10T01:46:45.7532034Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:45.7532550Z 2025-10-10T01:46:45.7533253Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:45.7534152Z 2025-10-10T01:46:45.7534567Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:45.7534945Z 2025-10-10T01:46:45.7535305Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7535665Z 2025-10-10T01:46:45.7536281Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:45.7536840Z 2025-10-10T01:46:45.7537250Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:45.7537649Z 2025-10-10T01:46:45.7538414Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:45.7539123Z 2025-10-10T01:46:45.7539589Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:45.7539950Z 2025-10-10T01:46:45.7540236Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7540515Z 2025-10-10T01:46:45.7541020Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:45.7541515Z 2025-10-10T01:46:45.7541767Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] self._init_executor() 2025-10-10T01:46:45.7542038Z 2025-10-10T01:46:45.7542872Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:45.7543763Z 2025-10-10T01:46:45.7544143Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:45.7544475Z 2025-10-10T01:46:45.7545443Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:45.7545969Z 2025-10-10T01:46:45.7546343Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:45.7546829Z 2025-10-10T01:46:45.7547124Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7547413Z 2025-10-10T01:46:45.7547978Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:45.7548447Z 2025-10-10T01:46:45.7548721Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:45.7548995Z 2025-10-10T01:46:45.7549242Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7549497Z 2025-10-10T01:46:45.7550014Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:45.7550490Z 2025-10-10T01:46:45.7550784Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:45.7551079Z 2025-10-10T01:46:45.7551355Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7551674Z 2025-10-10T01:46:45.7552201Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:45.7552685Z 2025-10-10T01:46:45.7552991Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:45.7553300Z 2025-10-10T01:46:45.7553572Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7553851Z 2025-10-10T01:46:45.7554293Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:45.7554706Z 2025-10-10T01:46:45.7555045Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:45.7555371Z 2025-10-10T01:46:45.7555703Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7555979Z 2025-10-10T01:46:45.7556324Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:45.7556651Z 2025-10-10T01:46:45.7557001Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:45.7557347Z 2025-10-10T01:46:45.7557716Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:45.7558078Z 2025-10-10T01:46:45.7558425Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:45.7558765Z 2025-10-10T01:46:45.7559269Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:45.7559625Z 2025-10-10T01:46:45.7559999Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:45.7560353Z 2025-10-10T01:46:45.7560852Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:45.7561377Z 2025-10-10T01:46:45.7561732Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:45.7562062Z 2025-10-10T01:46:45.7562622Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:45.7563103Z 2025-10-10T01:46:45.7563476Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:45.7563828Z 2025-10-10T01:46:45.7564388Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:45.7564905Z 2025-10-10T01:46:45.7565195Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:45.7565505Z 2025-10-10T01:46:45.7566139Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:45.7566759Z 2025-10-10T01:46:45.7567127Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:45.7567482Z 2025-10-10T01:46:45.7567732Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7567988Z 2025-10-10T01:46:45.7568549Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:45.7569063Z 2025-10-10T01:46:45.7569360Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:45.7569664Z 2025-10-10T01:46:45.7569942Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7570218Z 2025-10-10T01:46:45.7570860Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:45.7571431Z 2025-10-10T01:46:45.7571717Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:45.7572008Z 2025-10-10T01:46:45.7572266Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7572539Z 2025-10-10T01:46:45.7573122Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:45.7573657Z 2025-10-10T01:46:45.7573976Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:45.7574285Z 2025-10-10T01:46:45.7574561Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7574832Z 2025-10-10T01:46:45.7575358Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:45.7575836Z 2025-10-10T01:46:45.7576116Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:45.7576455Z 2025-10-10T01:46:45.7576720Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7576987Z 2025-10-10T01:46:45.7577540Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:45.7578031Z 2025-10-10T01:46:45.7578347Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:45.7578654Z 2025-10-10T01:46:45.7578874Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:45.7579120Z 2025-10-10T01:46:45.7579598Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:45.7580047Z 2025-10-10T01:46:45.7580297Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] raise RuntimeError( 2025-10-10T01:46:45.7580551Z 2025-10-10T01:46:45.7581124Z (EngineCore_DP0 pid=5633) ERROR 10-10 01:46:45 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:45.7581868Z (EngineCore_DP0 pid=5633) Process EngineCore_DP0: 2025-10-10T01:46:45.7582270Z (EngineCore_DP0 pid=5633) Traceback (most recent call last): 2025-10-10T01:46:45.7582898Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:45.7583414Z (EngineCore_DP0 pid=5633) self.run() 2025-10-10T01:46:45.7583940Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:45.7584503Z (EngineCore_DP0 pid=5633) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:45.7585167Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:45.7585707Z (EngineCore_DP0 pid=5633) raise e 2025-10-10T01:46:45.7586347Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:45.7586984Z (EngineCore_DP0 pid=5633) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:45.7587424Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7588047Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:45.7588672Z (EngineCore_DP0 pid=5633) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:45.7589324Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:45.7589936Z (EngineCore_DP0 pid=5633) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:45.7590390Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7591035Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:45.7591617Z (EngineCore_DP0 pid=5633) self._init_executor() 2025-10-10T01:46:45.7592267Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:45.7592947Z (EngineCore_DP0 pid=5633) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:45.7593738Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:45.7594484Z (EngineCore_DP0 pid=5633) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:45.7594984Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7595616Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:45.7596410Z (EngineCore_DP0 pid=5633) return func(*args, **kwargs) 2025-10-10T01:46:45.7596815Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7597450Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:45.7598080Z (EngineCore_DP0 pid=5633) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:45.7598508Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7599221Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:45.7599969Z (EngineCore_DP0 pid=5633) module = importlib.import_module(module_name) 2025-10-10T01:46:45.7600416Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7600993Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:45.7601611Z (EngineCore_DP0 pid=5633) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:45.7602103Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7602602Z (EngineCore_DP0 pid=5633) File "", line 1387, in _gcd_import 2025-10-10T01:46:45.7603159Z (EngineCore_DP0 pid=5633) File "", line 1360, in _find_and_load 2025-10-10T01:46:45.7603831Z (EngineCore_DP0 pid=5633) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:45.7604412Z (EngineCore_DP0 pid=5633) File "", line 935, in _load_unlocked 2025-10-10T01:46:45.7604982Z (EngineCore_DP0 pid=5633) File "", line 999, in exec_module 2025-10-10T01:46:45.7605582Z (EngineCore_DP0 pid=5633) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:45.7606311Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:45.7606995Z (EngineCore_DP0 pid=5633) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:45.7607710Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:45.7608427Z (EngineCore_DP0 pid=5633) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:45.7609206Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:45.7609887Z (EngineCore_DP0 pid=5633) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:45.7610676Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:45.7611574Z (EngineCore_DP0 pid=5633) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:45.7612048Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7612804Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:45.7613499Z (EngineCore_DP0 pid=5633) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:45.7613933Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7614668Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:45.7615355Z (EngineCore_DP0 pid=5633) return _is_fa2_supported(device)[0] 2025-10-10T01:46:45.7615762Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7616470Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:45.7617232Z (EngineCore_DP0 pid=5633) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:45.7617682Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7618346Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:45.7618982Z (EngineCore_DP0 pid=5633) prop = get_device_properties(device) 2025-10-10T01:46:45.7619393Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:45.7620043Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:45.7620718Z (EngineCore_DP0 pid=5633) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:45.7621111Z (EngineCore_DP0 pid=5633) ^^^^^^^^^^^^ 2025-10-10T01:46:45.7621759Z (EngineCore_DP0 pid=5633) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:45.7622320Z (EngineCore_DP0 pid=5633) raise RuntimeError( 2025-10-10T01:46:45.7622998Z (EngineCore_DP0 pid=5633) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:46.1663216Z FAILED 2025-10-10T01:46:46.1793013Z models/test_initialization.py::test_can_initialize_large_subset[GPT2LMHeadModel] Fork a new process to run a test 5637 2025-10-10T01:46:46.1804914Z Fork a new process to run a test 0 2025-10-10T01:46:46.2087028Z INFO 10-10 01:46:46 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GPT2LMHeadModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'openai-community/gpt2'} 2025-10-10T01:46:46.2970543Z 2025-10-10T01:46:46.2971529Z config.json: 0% 0.00/665 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:46:53.9074661Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] EngineCore failed to start. 2025-10-10T01:46:53.9075239Z 2025-10-10T01:46:53.9075663Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] Traceback (most recent call last): 2025-10-10T01:46:53.9076160Z 2025-10-10T01:46:53.9076771Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:53.9077278Z 2025-10-10T01:46:53.9077602Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:53.9078181Z 2025-10-10T01:46:53.9078466Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9078749Z 2025-10-10T01:46:53.9079500Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:53.9079972Z 2025-10-10T01:46:53.9080311Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:53.9080831Z 2025-10-10T01:46:53.9081617Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:53.9082198Z 2025-10-10T01:46:53.9082530Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:53.9082845Z 2025-10-10T01:46:53.9083132Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9083403Z 2025-10-10T01:46:53.9083907Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:53.9084490Z 2025-10-10T01:46:53.9084887Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] self._init_executor() 2025-10-10T01:46:53.9085313Z 2025-10-10T01:46:53.9086127Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:53.9086751Z 2025-10-10T01:46:53.9087182Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:53.9087527Z 2025-10-10T01:46:53.9088206Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:53.9088873Z 2025-10-10T01:46:53.9089455Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:53.9089932Z 2025-10-10T01:46:53.9090259Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9090557Z 2025-10-10T01:46:53.9091066Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:53.9091526Z 2025-10-10T01:46:53.9091796Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:46:53.9092071Z 2025-10-10T01:46:53.9092318Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9092580Z 2025-10-10T01:46:53.9093097Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:53.9093583Z 2025-10-10T01:46:53.9093869Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:53.9094159Z 2025-10-10T01:46:53.9094423Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9094691Z 2025-10-10T01:46:53.9095221Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:53.9095765Z 2025-10-10T01:46:53.9096383Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:46:53.9096764Z 2025-10-10T01:46:53.9097172Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9097476Z 2025-10-10T01:46:53.9097948Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:53.9098374Z 2025-10-10T01:46:53.9098731Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:53.9099076Z 2025-10-10T01:46:53.9099367Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9099657Z 2025-10-10T01:46:53.9100000Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:46:53.9100339Z 2025-10-10T01:46:53.9100692Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:46:53.9101107Z 2025-10-10T01:46:53.9101493Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:53.9101868Z 2025-10-10T01:46:53.9102224Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:46:53.9102558Z 2025-10-10T01:46:53.9102929Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:46:53.9103288Z 2025-10-10T01:46:53.9103660Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:53.9104026Z 2025-10-10T01:46:53.9104597Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:53.9105075Z 2025-10-10T01:46:53.9105434Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:53.9105784Z 2025-10-10T01:46:53.9106310Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:53.9106792Z 2025-10-10T01:46:53.9107160Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:53.9107518Z 2025-10-10T01:46:53.9108092Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:53.9108613Z 2025-10-10T01:46:53.9108914Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:46:53.9109209Z 2025-10-10T01:46:53.9109852Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:53.9110428Z 2025-10-10T01:46:53.9110801Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:53.9111223Z 2025-10-10T01:46:53.9111480Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9111749Z 2025-10-10T01:46:53.9112358Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:53.9112888Z 2025-10-10T01:46:53.9113188Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:46:53.9113489Z 2025-10-10T01:46:53.9113760Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9114039Z 2025-10-10T01:46:53.9114633Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:53.9115183Z 2025-10-10T01:46:53.9115468Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:46:53.9115750Z 2025-10-10T01:46:53.9116022Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9116333Z 2025-10-10T01:46:53.9116919Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:53.9117444Z 2025-10-10T01:46:53.9117761Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:53.9118078Z 2025-10-10T01:46:53.9118349Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9118628Z 2025-10-10T01:46:53.9119245Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:53.9119745Z 2025-10-10T01:46:53.9120030Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:46:53.9120321Z 2025-10-10T01:46:53.9120651Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9120921Z 2025-10-10T01:46:53.9121449Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:53.9121931Z 2025-10-10T01:46:53.9122243Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:46:53.9122547Z 2025-10-10T01:46:53.9122783Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:46:53.9123033Z 2025-10-10T01:46:53.9123515Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:53.9123970Z 2025-10-10T01:46:53.9124218Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] raise RuntimeError( 2025-10-10T01:46:53.9124479Z 2025-10-10T01:46:53.9125042Z (EngineCore_DP0 pid=5696) ERROR 10-10 01:46:53 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:53.9125736Z (EngineCore_DP0 pid=5696) Process EngineCore_DP0: 2025-10-10T01:46:53.9126162Z (EngineCore_DP0 pid=5696) Traceback (most recent call last): 2025-10-10T01:46:53.9126819Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:46:53.9127339Z (EngineCore_DP0 pid=5696) self.run() 2025-10-10T01:46:53.9127933Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:46:53.9128524Z (EngineCore_DP0 pid=5696) self._target(*self._args, **self._kwargs) 2025-10-10T01:46:53.9129198Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:46:53.9129763Z (EngineCore_DP0 pid=5696) raise e 2025-10-10T01:46:53.9130355Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:46:53.9130994Z (EngineCore_DP0 pid=5696) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:46:53.9131434Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9132058Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:46:53.9132763Z (EngineCore_DP0 pid=5696) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:46:53.9133426Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:46:53.9134044Z (EngineCore_DP0 pid=5696) self.model_executor = executor_class(vllm_config) 2025-10-10T01:46:53.9134492Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9135157Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:46:53.9135762Z (EngineCore_DP0 pid=5696) self._init_executor() 2025-10-10T01:46:53.9136437Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:46:53.9137171Z (EngineCore_DP0 pid=5696) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:46:53.9137897Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:46:53.9138608Z (EngineCore_DP0 pid=5696) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:46:53.9139105Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9139756Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:46:53.9140348Z (EngineCore_DP0 pid=5696) return func(*args, **kwargs) 2025-10-10T01:46:53.9140730Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9141378Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:46:53.9142022Z (EngineCore_DP0 pid=5696) worker_class = resolve_obj_by_qualname( 2025-10-10T01:46:53.9142449Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9143113Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:46:53.9143767Z (EngineCore_DP0 pid=5696) module = importlib.import_module(module_name) 2025-10-10T01:46:53.9144267Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9144850Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:46:53.9145521Z (EngineCore_DP0 pid=5696) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:46:53.9146013Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9146508Z (EngineCore_DP0 pid=5696) File "", line 1387, in _gcd_import 2025-10-10T01:46:53.9147071Z (EngineCore_DP0 pid=5696) File "", line 1360, in _find_and_load 2025-10-10T01:46:53.9147658Z (EngineCore_DP0 pid=5696) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:46:53.9148232Z (EngineCore_DP0 pid=5696) File "", line 935, in _load_unlocked 2025-10-10T01:46:53.9148805Z (EngineCore_DP0 pid=5696) File "", line 999, in exec_module 2025-10-10T01:46:53.9149410Z (EngineCore_DP0 pid=5696) File "", line 488, in _call_with_frames_removed 2025-10-10T01:46:53.9150205Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:46:53.9150884Z (EngineCore_DP0 pid=5696) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:46:53.9151600Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:46:53.9152317Z (EngineCore_DP0 pid=5696) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:46:53.9153097Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:46:53.9153768Z (EngineCore_DP0 pid=5696) class FlashAttentionMetadataBuilder( 2025-10-10T01:46:53.9154599Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:46:53.9155427Z (EngineCore_DP0 pid=5696) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:46:53.9155917Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9156601Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:46:53.9157281Z (EngineCore_DP0 pid=5696) if not is_fa_version_supported(fa_version): 2025-10-10T01:46:53.9157720Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9158461Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:46:53.9159216Z (EngineCore_DP0 pid=5696) return _is_fa2_supported(device)[0] 2025-10-10T01:46:53.9159624Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9160338Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:46:53.9161054Z (EngineCore_DP0 pid=5696) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:46:53.9161508Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9162247Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:46:53.9162889Z (EngineCore_DP0 pid=5696) prop = get_device_properties(device) 2025-10-10T01:46:53.9163346Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:46:53.9164013Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:46:53.9164678Z (EngineCore_DP0 pid=5696) _lazy_init() # will define _get_device_properties 2025-10-10T01:46:53.9165076Z (EngineCore_DP0 pid=5696) ^^^^^^^^^^^^ 2025-10-10T01:46:53.9165660Z (EngineCore_DP0 pid=5696) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:46:53.9166224Z (EngineCore_DP0 pid=5696) raise RuntimeError( 2025-10-10T01:46:53.9166905Z (EngineCore_DP0 pid=5696) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:46:54.3051071Z FAILED 2025-10-10T01:46:54.3179541Z models/test_initialization.py::test_can_initialize_large_subset[AriaForConditionalGeneration] Fork a new process to run a test 5700 2025-10-10T01:46:54.3191308Z Fork a new process to run a test 0 2025-10-10T01:46:54.3468621Z INFO 10-10 01:46:54 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='AriaForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'rhymes-ai/Aria'} 2025-10-10T01:46:54.4652139Z 2025-10-10T01:46:54.4654314Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:54.4654735Z config.json: 1.08kB [00:00, 5.55MB/s] 2025-10-10T01:46:54.5768709Z 2025-10-10T01:46:54.5770752Z preprocessor_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:46:54.5771115Z preprocessor_config.json: 1.21kB [00:00, 4.56MB/s] 2025-10-10T01:47:01.2145476Z INFO 10-10 01:47:01 [model.py:551] Resolved architecture: AriaForConditionalGeneration 2025-10-10T01:47:01.2146001Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:47:01.2389517Z INFO 10-10 01:47:01 [model.py:1545] Using max model len 65536 2025-10-10T01:47:01.4061317Z INFO 10-10 01:47:01 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:47:01.4446839Z 2025-10-10T01:47:01.4452278Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:47:01.4452647Z tokenizer_config.json: 110kB [00:00, 211MB/s] 2025-10-10T01:47:01.5535469Z 2025-10-10T01:47:01.8417749Z tokenizer.model: 0% 0.00/1.70M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:04.7048750Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:04.7049346Z 2025-10-10T01:47:04.7049903Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:04.7050447Z 2025-10-10T01:47:04.7051642Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:04.7052549Z 2025-10-10T01:47:04.7053124Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:04.7053670Z 2025-10-10T01:47:04.7054175Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7054689Z 2025-10-10T01:47:04.7055575Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:04.7056439Z 2025-10-10T01:47:04.7057094Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:04.7057723Z 2025-10-10T01:47:04.7058691Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:04.7059554Z 2025-10-10T01:47:04.7060199Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:04.7060797Z 2025-10-10T01:47:04.7061371Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7061917Z 2025-10-10T01:47:04.7063138Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:04.7064037Z 2025-10-10T01:47:04.7064716Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] self._init_executor() 2025-10-10T01:47:04.7065232Z 2025-10-10T01:47:04.7066295Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:04.7067247Z 2025-10-10T01:47:04.7067811Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:04.7068372Z 2025-10-10T01:47:04.7069357Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:04.7070134Z 2025-10-10T01:47:04.7070521Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:04.7070880Z 2025-10-10T01:47:04.7071180Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7071620Z 2025-10-10T01:47:04.7072136Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:04.7072595Z 2025-10-10T01:47:04.7072865Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:04.7073133Z 2025-10-10T01:47:04.7073401Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7073669Z 2025-10-10T01:47:04.7074206Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:04.7074710Z 2025-10-10T01:47:04.7075004Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:04.7075300Z 2025-10-10T01:47:04.7075622Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7075910Z 2025-10-10T01:47:04.7076449Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:04.7076932Z 2025-10-10T01:47:04.7077243Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:04.7077549Z 2025-10-10T01:47:04.7077831Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7078102Z 2025-10-10T01:47:04.7078553Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:04.7078964Z 2025-10-10T01:47:04.7079470Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:04.7079800Z 2025-10-10T01:47:04.7080085Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7080368Z 2025-10-10T01:47:04.7080707Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:04.7081100Z 2025-10-10T01:47:04.7081447Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:04.7081790Z 2025-10-10T01:47:04.7082214Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:04.7082581Z 2025-10-10T01:47:04.7082940Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:04.7083270Z 2025-10-10T01:47:04.7083636Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:04.7083983Z 2025-10-10T01:47:04.7084363Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:04.7084723Z 2025-10-10T01:47:04.7085228Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:04.7085686Z 2025-10-10T01:47:04.7086045Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:04.7086438Z 2025-10-10T01:47:04.7086960Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:04.7087444Z 2025-10-10T01:47:04.7087812Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:04.7088172Z 2025-10-10T01:47:04.7088721Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:04.7089231Z 2025-10-10T01:47:04.7089521Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:04.7089817Z 2025-10-10T01:47:04.7090567Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:04.7091153Z 2025-10-10T01:47:04.7091530Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:04.7091881Z 2025-10-10T01:47:04.7092157Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7092417Z 2025-10-10T01:47:04.7092977Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:04.7093499Z 2025-10-10T01:47:04.7093792Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:04.7094091Z 2025-10-10T01:47:04.7094363Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7094649Z 2025-10-10T01:47:04.7095246Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:04.7095802Z 2025-10-10T01:47:04.7096306Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:04.7096744Z 2025-10-10T01:47:04.7097034Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7097303Z 2025-10-10T01:47:04.7097980Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:04.7098542Z 2025-10-10T01:47:04.7098870Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:04.7099178Z 2025-10-10T01:47:04.7099456Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7099738Z 2025-10-10T01:47:04.7100266Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:04.7100768Z 2025-10-10T01:47:04.7101052Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:04.7101341Z 2025-10-10T01:47:04.7101602Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7101940Z 2025-10-10T01:47:04.7102465Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:04.7102940Z 2025-10-10T01:47:04.7103252Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:04.7103553Z 2025-10-10T01:47:04.7103790Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:04.7104032Z 2025-10-10T01:47:04.7104518Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:04.7104965Z 2025-10-10T01:47:04.7105209Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] raise RuntimeError( 2025-10-10T01:47:04.7105471Z 2025-10-10T01:47:04.7106112Z (EngineCore_DP0 pid=5779) ERROR 10-10 01:47:04 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:04.7106840Z (EngineCore_DP0 pid=5779) Process EngineCore_DP0: 2025-10-10T01:47:04.7107252Z (EngineCore_DP0 pid=5779) Traceback (most recent call last): 2025-10-10T01:47:04.7107861Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:04.7108370Z (EngineCore_DP0 pid=5779) self.run() 2025-10-10T01:47:04.7108898Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:04.7109454Z (EngineCore_DP0 pid=5779) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:04.7110125Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:04.7110681Z (EngineCore_DP0 pid=5779) raise e 2025-10-10T01:47:04.7111271Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:04.7111910Z (EngineCore_DP0 pid=5779) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:04.7112349Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7113022Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:04.7113657Z (EngineCore_DP0 pid=5779) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:04.7114350Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:04.7115001Z (EngineCore_DP0 pid=5779) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:04.7115455Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7116126Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:04.7116711Z (EngineCore_DP0 pid=5779) self._init_executor() 2025-10-10T01:47:04.7117369Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:04.7118054Z (EngineCore_DP0 pid=5779) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:04.7118778Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:04.7119667Z (EngineCore_DP0 pid=5779) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:04.7120162Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7120802Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:04.7121390Z (EngineCore_DP0 pid=5779) return func(*args, **kwargs) 2025-10-10T01:47:04.7121772Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7122412Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:04.7123044Z (EngineCore_DP0 pid=5779) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:04.7123513Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7124194Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:04.7124845Z (EngineCore_DP0 pid=5779) module = importlib.import_module(module_name) 2025-10-10T01:47:04.7125287Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7125866Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:04.7126508Z (EngineCore_DP0 pid=5779) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:04.7127001Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7127493Z (EngineCore_DP0 pid=5779) File "", line 1387, in _gcd_import 2025-10-10T01:47:04.7128049Z (EngineCore_DP0 pid=5779) File "", line 1360, in _find_and_load 2025-10-10T01:47:04.7128641Z (EngineCore_DP0 pid=5779) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:04.7129220Z (EngineCore_DP0 pid=5779) File "", line 935, in _load_unlocked 2025-10-10T01:47:04.7129787Z (EngineCore_DP0 pid=5779) File "", line 999, in exec_module 2025-10-10T01:47:04.7130440Z (EngineCore_DP0 pid=5779) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:04.7131170Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:04.7131890Z (EngineCore_DP0 pid=5779) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:04.7132612Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:04.7133336Z (EngineCore_DP0 pid=5779) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:04.7134109Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:04.7134770Z (EngineCore_DP0 pid=5779) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:04.7135556Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:04.7136380Z (EngineCore_DP0 pid=5779) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:04.7136902Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7137592Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:04.7138260Z (EngineCore_DP0 pid=5779) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:04.7138692Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7139431Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:04.7140128Z (EngineCore_DP0 pid=5779) return _is_fa2_supported(device)[0] 2025-10-10T01:47:04.7140533Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7141280Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:04.7141994Z (EngineCore_DP0 pid=5779) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:04.7142445Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7143107Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:04.7143741Z (EngineCore_DP0 pid=5779) prop = get_device_properties(device) 2025-10-10T01:47:04.7144154Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:04.7144809Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:04.7145468Z (EngineCore_DP0 pid=5779) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:04.7145860Z (EngineCore_DP0 pid=5779) ^^^^^^^^^^^^ 2025-10-10T01:47:04.7146436Z (EngineCore_DP0 pid=5779) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:04.7146992Z (EngineCore_DP0 pid=5779) raise RuntimeError( 2025-10-10T01:47:04.7147664Z (EngineCore_DP0 pid=5779) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:05.1259482Z FAILED 2025-10-10T01:47:05.1387926Z models/test_initialization.py::test_can_initialize_large_subset[OPTForCausalLM] Fork a new process to run a test 5783 2025-10-10T01:47:05.1399164Z Fork a new process to run a test 0 2025-10-10T01:47:05.1673057Z INFO 10-10 01:47:05 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='OPTForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'facebook/opt-125m'} 2025-10-10T01:47:05.2519639Z 2025-10-10T01:47:05.2520567Z config.json: 0% 0.00/651 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:12.7286899Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:12.7287351Z 2025-10-10T01:47:12.7287724Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:12.7288084Z 2025-10-10T01:47:12.7288742Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:12.7289341Z 2025-10-10T01:47:12.7289737Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:12.7290129Z 2025-10-10T01:47:12.7290479Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7290816Z 2025-10-10T01:47:12.7291770Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:12.7292540Z 2025-10-10T01:47:12.7293079Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:12.7293511Z 2025-10-10T01:47:12.7294001Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:12.7294444Z 2025-10-10T01:47:12.7294769Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:12.7295079Z 2025-10-10T01:47:12.7295558Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7296274Z 2025-10-10T01:47:12.7297497Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:12.7298407Z 2025-10-10T01:47:12.7298939Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] self._init_executor() 2025-10-10T01:47:12.7299440Z 2025-10-10T01:47:12.7300109Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:12.7300610Z 2025-10-10T01:47:12.7300947Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:12.7301261Z 2025-10-10T01:47:12.7301800Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:12.7302291Z 2025-10-10T01:47:12.7302641Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:12.7302982Z 2025-10-10T01:47:12.7303271Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7303555Z 2025-10-10T01:47:12.7304044Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:12.7304631Z 2025-10-10T01:47:12.7304900Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:12.7305170Z 2025-10-10T01:47:12.7305415Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7305749Z 2025-10-10T01:47:12.7306277Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:12.7306749Z 2025-10-10T01:47:12.7307038Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:12.7307329Z 2025-10-10T01:47:12.7307598Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7307875Z 2025-10-10T01:47:12.7308427Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:12.7308931Z 2025-10-10T01:47:12.7309236Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:12.7309610Z 2025-10-10T01:47:12.7309887Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7310164Z 2025-10-10T01:47:12.7310599Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:12.7311010Z 2025-10-10T01:47:12.7311354Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:12.7311682Z 2025-10-10T01:47:12.7311972Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7312246Z 2025-10-10T01:47:12.7312590Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:12.7312921Z 2025-10-10T01:47:12.7313362Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:12.7313704Z 2025-10-10T01:47:12.7314076Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:12.7314451Z 2025-10-10T01:47:12.7314794Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:12.7315128Z 2025-10-10T01:47:12.7315487Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:12.7315840Z 2025-10-10T01:47:12.7316213Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:12.7316570Z 2025-10-10T01:47:12.7317078Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:12.7317538Z 2025-10-10T01:47:12.7317900Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:12.7318231Z 2025-10-10T01:47:12.7318751Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:12.7319411Z 2025-10-10T01:47:12.7319794Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:12.7320151Z 2025-10-10T01:47:12.7320744Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:12.7321283Z 2025-10-10T01:47:12.7321586Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:12.7321888Z 2025-10-10T01:47:12.7322521Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:12.7323111Z 2025-10-10T01:47:12.7323482Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:12.7323842Z 2025-10-10T01:47:12.7324097Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7324399Z 2025-10-10T01:47:12.7324975Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:12.7325489Z 2025-10-10T01:47:12.7325791Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:12.7326081Z 2025-10-10T01:47:12.7326360Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7326631Z 2025-10-10T01:47:12.7327234Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:12.7327782Z 2025-10-10T01:47:12.7328061Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:12.7328355Z 2025-10-10T01:47:12.7328649Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7328926Z 2025-10-10T01:47:12.7329500Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:12.7330033Z 2025-10-10T01:47:12.7330348Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:12.7330654Z 2025-10-10T01:47:12.7330931Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7331202Z 2025-10-10T01:47:12.7331739Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:12.7332223Z 2025-10-10T01:47:12.7332508Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:12.7332789Z 2025-10-10T01:47:12.7333053Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7333319Z 2025-10-10T01:47:12.7333835Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:12.7334363Z 2025-10-10T01:47:12.7334672Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:12.7334981Z 2025-10-10T01:47:12.7335201Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:12.7335444Z 2025-10-10T01:47:12.7335956Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:12.7336401Z 2025-10-10T01:47:12.7336651Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] raise RuntimeError( 2025-10-10T01:47:12.7336906Z 2025-10-10T01:47:12.7337489Z (EngineCore_DP0 pid=5841) ERROR 10-10 01:47:12 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:12.7338186Z (EngineCore_DP0 pid=5841) Process EngineCore_DP0: 2025-10-10T01:47:12.7338597Z (EngineCore_DP0 pid=5841) Traceback (most recent call last): 2025-10-10T01:47:12.7339191Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:12.7339764Z (EngineCore_DP0 pid=5841) self.run() 2025-10-10T01:47:12.7340288Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:12.7340847Z (EngineCore_DP0 pid=5841) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:12.7341510Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:12.7342050Z (EngineCore_DP0 pid=5841) raise e 2025-10-10T01:47:12.7342646Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:12.7343275Z (EngineCore_DP0 pid=5841) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:12.7343713Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7344377Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:12.7345011Z (EngineCore_DP0 pid=5841) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:12.7345663Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:12.7346277Z (EngineCore_DP0 pid=5841) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:12.7346727Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7347381Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:12.7347967Z (EngineCore_DP0 pid=5841) self._init_executor() 2025-10-10T01:47:12.7348637Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:12.7349318Z (EngineCore_DP0 pid=5841) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:12.7350040Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:12.7350749Z (EngineCore_DP0 pid=5841) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:12.7351243Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7351924Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:12.7352513Z (EngineCore_DP0 pid=5841) return func(*args, **kwargs) 2025-10-10T01:47:12.7352936Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7353576Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:12.7354205Z (EngineCore_DP0 pid=5841) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:12.7354619Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7355298Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:12.7355951Z (EngineCore_DP0 pid=5841) module = importlib.import_module(module_name) 2025-10-10T01:47:12.7356395Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7356976Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:12.7357638Z (EngineCore_DP0 pid=5841) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:12.7358127Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7358618Z (EngineCore_DP0 pid=5841) File "", line 1387, in _gcd_import 2025-10-10T01:47:12.7359223Z (EngineCore_DP0 pid=5841) File "", line 1360, in _find_and_load 2025-10-10T01:47:12.7359805Z (EngineCore_DP0 pid=5841) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:12.7360372Z (EngineCore_DP0 pid=5841) File "", line 935, in _load_unlocked 2025-10-10T01:47:12.7360941Z (EngineCore_DP0 pid=5841) File "", line 999, in exec_module 2025-10-10T01:47:12.7361587Z (EngineCore_DP0 pid=5841) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:12.7362327Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:12.7363002Z (EngineCore_DP0 pid=5841) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:12.7363716Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:12.7364425Z (EngineCore_DP0 pid=5841) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:12.7365201Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:12.7365873Z (EngineCore_DP0 pid=5841) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:12.7366660Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:12.7367468Z (EngineCore_DP0 pid=5841) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:12.7367935Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7368619Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:12.7369357Z (EngineCore_DP0 pid=5841) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:12.7369789Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7370567Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:12.7371268Z (EngineCore_DP0 pid=5841) return _is_fa2_supported(device)[0] 2025-10-10T01:47:12.7371672Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7372382Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:12.7373094Z (EngineCore_DP0 pid=5841) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:12.7373548Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7374212Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:12.7374899Z (EngineCore_DP0 pid=5841) prop = get_device_properties(device) 2025-10-10T01:47:12.7375313Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:12.7375964Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:12.7376619Z (EngineCore_DP0 pid=5841) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:12.7377028Z (EngineCore_DP0 pid=5841) ^^^^^^^^^^^^ 2025-10-10T01:47:12.7377601Z (EngineCore_DP0 pid=5841) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:12.7378161Z (EngineCore_DP0 pid=5841) raise RuntimeError( 2025-10-10T01:47:12.7378841Z (EngineCore_DP0 pid=5841) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:13.1279511Z FAILED 2025-10-10T01:47:13.1408430Z models/test_initialization.py::test_can_initialize_large_subset[BambaForCausalLM] Fork a new process to run a test 5845 2025-10-10T01:47:13.1420120Z Fork a new process to run a test 0 2025-10-10T01:47:13.1702431Z INFO 10-10 01:47:13 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BambaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ibm-ai-platform/Bamba-9B-v1'} 2025-10-10T01:47:13.2844255Z 2025-10-10T01:47:13.2845032Z config.json: 0% 0.00/921 [00:00= mamba page size. 2025-10-10T01:47:20.5035971Z INFO 10-10 01:47:20 [config.py:397] Padding mamba page size by 0.69% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:47:20.5474944Z 2025-10-10T01:47:20.5477970Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:47:20.5478319Z tokenizer_config.json: 50.6kB [00:00, 190MB/s] 2025-10-10T01:47:20.6257029Z 2025-10-10T01:47:20.6577907Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:47:20.6578219Z tokenizer.json: 9.09MB [00:00, 283MB/s] 2025-10-10T01:47:20.7443329Z 2025-10-10T01:47:20.7443986Z special_tokens_map.json: 0% 0.00/301 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:21.4395562Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:21.4395978Z 2025-10-10T01:47:21.4396618Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:21.4396988Z 2025-10-10T01:47:21.4397667Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:21.4398526Z 2025-10-10T01:47:21.4398940Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:21.4399469Z 2025-10-10T01:47:21.4399953Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4400325Z 2025-10-10T01:47:21.4400960Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:21.4401510Z 2025-10-10T01:47:21.4401850Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:21.4402185Z 2025-10-10T01:47:21.4402684Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:21.4403154Z 2025-10-10T01:47:21.4403494Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:21.4403808Z 2025-10-10T01:47:21.4404089Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4404477Z 2025-10-10T01:47:21.4405002Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:21.4405476Z 2025-10-10T01:47:21.4405734Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] self._init_executor() 2025-10-10T01:47:21.4406008Z 2025-10-10T01:47:21.4406557Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:21.4407072Z 2025-10-10T01:47:21.4407421Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:21.4407744Z 2025-10-10T01:47:21.4408387Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:21.4408902Z 2025-10-10T01:47:21.4409255Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:21.4409586Z 2025-10-10T01:47:21.4409883Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4410167Z 2025-10-10T01:47:21.4410661Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:21.4411118Z 2025-10-10T01:47:21.4411393Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:21.4411661Z 2025-10-10T01:47:21.4411919Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4412179Z 2025-10-10T01:47:21.4412721Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:21.4413212Z 2025-10-10T01:47:21.4413530Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:21.4413821Z 2025-10-10T01:47:21.4414102Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4414421Z 2025-10-10T01:47:21.4414984Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:21.4415472Z 2025-10-10T01:47:21.4415814Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:21.4416134Z 2025-10-10T01:47:21.4416417Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4416705Z 2025-10-10T01:47:21.4417151Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:21.4417575Z 2025-10-10T01:47:21.4417912Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:21.4418252Z 2025-10-10T01:47:21.4418553Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4418831Z 2025-10-10T01:47:21.4419184Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:21.4419567Z 2025-10-10T01:47:21.4419932Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:21.4420272Z 2025-10-10T01:47:21.4420648Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:21.4421020Z 2025-10-10T01:47:21.4421360Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:21.4421698Z 2025-10-10T01:47:21.4422057Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:21.4422427Z 2025-10-10T01:47:21.4422813Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:21.4423183Z 2025-10-10T01:47:21.4423721Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:21.4424187Z 2025-10-10T01:47:21.4424550Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:21.4424884Z 2025-10-10T01:47:21.4425425Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:21.4425906Z 2025-10-10T01:47:21.4426276Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:21.4426635Z 2025-10-10T01:47:21.4427199Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:21.4427739Z 2025-10-10T01:47:21.4428041Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:21.4428344Z 2025-10-10T01:47:21.4428978Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:21.4429611Z 2025-10-10T01:47:21.4429989Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:21.4430346Z 2025-10-10T01:47:21.4430639Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4430924Z 2025-10-10T01:47:21.4431504Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:21.4432020Z 2025-10-10T01:47:21.4432330Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:21.4432622Z 2025-10-10T01:47:21.4432897Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4433170Z 2025-10-10T01:47:21.4433773Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:21.4434323Z 2025-10-10T01:47:21.4434674Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:21.4434962Z 2025-10-10T01:47:21.4435223Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4435495Z 2025-10-10T01:47:21.4436084Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:21.4436617Z 2025-10-10T01:47:21.4436941Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:21.4437259Z 2025-10-10T01:47:21.4437529Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4437802Z 2025-10-10T01:47:21.4438343Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:21.4438870Z 2025-10-10T01:47:21.4439264Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:21.4439553Z 2025-10-10T01:47:21.4439834Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4440101Z 2025-10-10T01:47:21.4440645Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:21.4441131Z 2025-10-10T01:47:21.4441449Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:21.4441756Z 2025-10-10T01:47:21.4441981Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:21.4442230Z 2025-10-10T01:47:21.4442714Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:21.4443168Z 2025-10-10T01:47:21.4443409Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] raise RuntimeError( 2025-10-10T01:47:21.4443659Z 2025-10-10T01:47:21.4444224Z (EngineCore_DP0 pid=5907) ERROR 10-10 01:47:21 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:21.4444993Z (EngineCore_DP0 pid=5907) Process EngineCore_DP0: 2025-10-10T01:47:21.4445400Z (EngineCore_DP0 pid=5907) Traceback (most recent call last): 2025-10-10T01:47:21.4446058Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:21.4446595Z (EngineCore_DP0 pid=5907) self.run() 2025-10-10T01:47:21.4447138Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:21.4447699Z (EngineCore_DP0 pid=5907) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:21.4448364Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:21.4448918Z (EngineCore_DP0 pid=5907) raise e 2025-10-10T01:47:21.4449514Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:21.4450156Z (EngineCore_DP0 pid=5907) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:21.4450681Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4451302Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:21.4451957Z (EngineCore_DP0 pid=5907) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:21.4452615Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:21.4453243Z (EngineCore_DP0 pid=5907) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:21.4453705Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4454359Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:21.4454967Z (EngineCore_DP0 pid=5907) self._init_executor() 2025-10-10T01:47:21.4455685Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:21.4456384Z (EngineCore_DP0 pid=5907) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:21.4457111Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:21.4457823Z (EngineCore_DP0 pid=5907) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:21.4458318Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4458965Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:21.4459568Z (EngineCore_DP0 pid=5907) return func(*args, **kwargs) 2025-10-10T01:47:21.4459965Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4460610Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:21.4461256Z (EngineCore_DP0 pid=5907) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:21.4461683Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4462341Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:21.4463060Z (EngineCore_DP0 pid=5907) module = importlib.import_module(module_name) 2025-10-10T01:47:21.4463534Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4464170Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:21.4464820Z (EngineCore_DP0 pid=5907) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:21.4465318Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4465829Z (EngineCore_DP0 pid=5907) File "", line 1387, in _gcd_import 2025-10-10T01:47:21.4466397Z (EngineCore_DP0 pid=5907) File "", line 1360, in _find_and_load 2025-10-10T01:47:21.4466990Z (EngineCore_DP0 pid=5907) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:21.4467566Z (EngineCore_DP0 pid=5907) File "", line 935, in _load_unlocked 2025-10-10T01:47:21.4468198Z (EngineCore_DP0 pid=5907) File "", line 999, in exec_module 2025-10-10T01:47:21.4468816Z (EngineCore_DP0 pid=5907) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:21.4469553Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:21.4470239Z (EngineCore_DP0 pid=5907) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:21.4470968Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:21.4471696Z (EngineCore_DP0 pid=5907) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:21.4472484Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:21.4473211Z (EngineCore_DP0 pid=5907) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:21.4474020Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:21.4474842Z (EngineCore_DP0 pid=5907) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:21.4475320Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4475998Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:21.4476682Z (EngineCore_DP0 pid=5907) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:21.4477133Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4477896Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:21.4478593Z (EngineCore_DP0 pid=5907) return _is_fa2_supported(device)[0] 2025-10-10T01:47:21.4478995Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4479775Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:21.4480557Z (EngineCore_DP0 pid=5907) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:21.4481014Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4481734Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:21.4482396Z (EngineCore_DP0 pid=5907) prop = get_device_properties(device) 2025-10-10T01:47:21.4482810Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:21.4483468Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:21.4484124Z (EngineCore_DP0 pid=5907) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:21.4484516Z (EngineCore_DP0 pid=5907) ^^^^^^^^^^^^ 2025-10-10T01:47:21.4485089Z (EngineCore_DP0 pid=5907) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:21.4485668Z (EngineCore_DP0 pid=5907) raise RuntimeError( 2025-10-10T01:47:21.4486357Z (EngineCore_DP0 pid=5907) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:21.8481517Z FAILED 2025-10-10T01:47:21.8609928Z models/test_initialization.py::test_can_initialize_large_subset[Cohere2VisionForConditionalGeneration] Fork a new process to run a test 5911 2025-10-10T01:47:21.8621410Z Fork a new process to run a test 0 2025-10-10T01:47:21.8891331Z INFO 10-10 01:47:21 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Cohere2VisionForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'CohereLabs/command-a-vision-07-2025'} 2025-10-10T01:47:22.0263713Z 2025-10-10T01:47:22.0264798Z config.json: 0% 0.00/3.59k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:31.4743699Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:31.4744311Z 2025-10-10T01:47:31.4744802Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:31.4745255Z 2025-10-10T01:47:31.4745958Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:31.4746565Z 2025-10-10T01:47:31.4746952Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:31.4747336Z 2025-10-10T01:47:31.4747673Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4748018Z 2025-10-10T01:47:31.4748631Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:31.4749195Z 2025-10-10T01:47:31.4749599Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:31.4750181Z 2025-10-10T01:47:31.4751006Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:31.4751537Z 2025-10-10T01:47:31.4751883Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:31.4752325Z 2025-10-10T01:47:31.4752621Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4752899Z 2025-10-10T01:47:31.4753502Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:31.4754115Z 2025-10-10T01:47:31.4754389Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] self._init_executor() 2025-10-10T01:47:31.4754772Z 2025-10-10T01:47:31.4755444Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:31.4762938Z 2025-10-10T01:47:31.4763411Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:31.4763783Z 2025-10-10T01:47:31.4764354Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:31.4765013Z 2025-10-10T01:47:31.4765378Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:31.4765717Z 2025-10-10T01:47:31.4766010Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4766291Z 2025-10-10T01:47:31.4766805Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:31.4767268Z 2025-10-10T01:47:31.4767548Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:31.4767819Z 2025-10-10T01:47:31.4768077Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4768336Z 2025-10-10T01:47:31.4768934Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:31.4769443Z 2025-10-10T01:47:31.4769737Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:31.4770034Z 2025-10-10T01:47:31.4770309Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4770586Z 2025-10-10T01:47:31.4771113Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:31.4771614Z 2025-10-10T01:47:31.4771933Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:31.4772242Z 2025-10-10T01:47:31.4772525Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4772798Z 2025-10-10T01:47:31.4773239Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:31.4773648Z 2025-10-10T01:47:31.4774036Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:31.4774374Z 2025-10-10T01:47:31.4774659Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4774992Z 2025-10-10T01:47:31.4775343Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:31.4775681Z 2025-10-10T01:47:31.4776084Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:31.4776418Z 2025-10-10T01:47:31.4776799Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:31.4777155Z 2025-10-10T01:47:31.4777497Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:31.4777839Z 2025-10-10T01:47:31.4778198Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:31.4778572Z 2025-10-10T01:47:31.4778948Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:31.4779360Z 2025-10-10T01:47:31.4779866Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:31.4780342Z 2025-10-10T01:47:31.4780692Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:31.4781030Z 2025-10-10T01:47:31.4781553Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:31.4782032Z 2025-10-10T01:47:31.4782407Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:31.4782764Z 2025-10-10T01:47:31.4783333Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:31.4783894Z 2025-10-10T01:47:31.4784203Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:31.4784499Z 2025-10-10T01:47:31.4785129Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:31.4785726Z 2025-10-10T01:47:31.4786108Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:31.4786470Z 2025-10-10T01:47:31.4786725Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4786991Z 2025-10-10T01:47:31.4787566Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:31.4788095Z 2025-10-10T01:47:31.4788394Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:31.4788686Z 2025-10-10T01:47:31.4788961Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4789230Z 2025-10-10T01:47:31.4789863Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:31.4790575Z 2025-10-10T01:47:31.4790917Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:31.4791254Z 2025-10-10T01:47:31.4791591Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4791882Z 2025-10-10T01:47:31.4792471Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:31.4793008Z 2025-10-10T01:47:31.4793337Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:31.4793660Z 2025-10-10T01:47:31.4793934Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4794216Z 2025-10-10T01:47:31.4794742Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:31.4795275Z 2025-10-10T01:47:31.4795566Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:31.4795855Z 2025-10-10T01:47:31.4796428Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4796720Z 2025-10-10T01:47:31.4797274Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:31.4797758Z 2025-10-10T01:47:31.4798071Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:31.4798387Z 2025-10-10T01:47:31.4798612Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:31.4798861Z 2025-10-10T01:47:31.4799529Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:31.4800003Z 2025-10-10T01:47:31.4800264Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] raise RuntimeError( 2025-10-10T01:47:31.4800543Z 2025-10-10T01:47:31.4801108Z (EngineCore_DP0 pid=5989) ERROR 10-10 01:47:31 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:31.4801804Z (EngineCore_DP0 pid=5989) Process EngineCore_DP0: 2025-10-10T01:47:31.4802210Z (EngineCore_DP0 pid=5989) Traceback (most recent call last): 2025-10-10T01:47:31.4802814Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:31.4803330Z (EngineCore_DP0 pid=5989) self.run() 2025-10-10T01:47:31.4803869Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:31.4804436Z (EngineCore_DP0 pid=5989) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:31.4805111Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:31.4805655Z (EngineCore_DP0 pid=5989) raise e 2025-10-10T01:47:31.4806244Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:31.4806956Z (EngineCore_DP0 pid=5989) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:31.4807400Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4808108Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:31.4808755Z (EngineCore_DP0 pid=5989) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:31.4809450Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:31.4810123Z (EngineCore_DP0 pid=5989) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:31.4810661Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4811443Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:31.4812131Z (EngineCore_DP0 pid=5989) self._init_executor() 2025-10-10T01:47:31.4812823Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:31.4813584Z (EngineCore_DP0 pid=5989) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:31.4814312Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:31.4815026Z (EngineCore_DP0 pid=5989) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:31.4815521Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4816184Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:31.4816772Z (EngineCore_DP0 pid=5989) return func(*args, **kwargs) 2025-10-10T01:47:31.4817165Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4817846Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:31.4818482Z (EngineCore_DP0 pid=5989) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:31.4818904Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4819571Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:31.4820323Z (EngineCore_DP0 pid=5989) module = importlib.import_module(module_name) 2025-10-10T01:47:31.4820854Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4821579Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:31.4822494Z (EngineCore_DP0 pid=5989) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:31.4823000Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4823508Z (EngineCore_DP0 pid=5989) File "", line 1387, in _gcd_import 2025-10-10T01:47:31.4824067Z (EngineCore_DP0 pid=5989) File "", line 1360, in _find_and_load 2025-10-10T01:47:31.4824661Z (EngineCore_DP0 pid=5989) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:31.4825300Z (EngineCore_DP0 pid=5989) File "", line 935, in _load_unlocked 2025-10-10T01:47:31.4825874Z (EngineCore_DP0 pid=5989) File "", line 999, in exec_module 2025-10-10T01:47:31.4826525Z (EngineCore_DP0 pid=5989) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:31.4827272Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:31.4827964Z (EngineCore_DP0 pid=5989) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:31.4828701Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:31.4829432Z (EngineCore_DP0 pid=5989) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:31.4830221Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:31.4830903Z (EngineCore_DP0 pid=5989) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:31.4831750Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:31.4832565Z (EngineCore_DP0 pid=5989) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:31.4833041Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4833739Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:31.4834440Z (EngineCore_DP0 pid=5989) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:31.4834896Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4835675Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:31.4836421Z (EngineCore_DP0 pid=5989) return _is_fa2_supported(device)[0] 2025-10-10T01:47:31.4836846Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4837565Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:31.4838302Z (EngineCore_DP0 pid=5989) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:31.4838768Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4839559Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:31.4840203Z (EngineCore_DP0 pid=5989) prop = get_device_properties(device) 2025-10-10T01:47:31.4840627Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:31.4841287Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:31.4841945Z (EngineCore_DP0 pid=5989) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:31.4842346Z (EngineCore_DP0 pid=5989) ^^^^^^^^^^^^ 2025-10-10T01:47:31.4842925Z (EngineCore_DP0 pid=5989) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:31.4843560Z (EngineCore_DP0 pid=5989) raise RuntimeError( 2025-10-10T01:47:31.4844246Z (EngineCore_DP0 pid=5989) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:31.9180284Z FAILED 2025-10-10T01:47:31.9309433Z models/test_initialization.py::test_can_initialize_large_subset[HunYuanMoEV1ForCausalLM] Fork a new process to run a test 5993 2025-10-10T01:47:31.9320738Z Fork a new process to run a test 0 2025-10-10T01:47:31.9596589Z INFO 10-10 01:47:31 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='HunYuanMoEV1ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'tencent/Hunyuan-A13B-Instruct'} 2025-10-10T01:47:32.0929306Z 2025-10-10T01:47:32.0931418Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:47:32.0931735Z config.json: 3.38kB [00:00, 17.5MB/s] 2025-10-10T01:47:32.1770763Z 2025-10-10T01:47:32.1773446Z configuration_hunyuan.py: 0.00B [00:00, ?B/s] 2025-10-10T01:47:32.1773962Z configuration_hunyuan.py: 15.0kB [00:00, 53.4MB/s] 2025-10-10T01:47:32.1880270Z A new version of the following files was downloaded from https://huggingface.co/tencent/Hunyuan-A13B-Instruct: 2025-10-10T01:47:32.1880929Z - configuration_hunyuan.py 2025-10-10T01:47:32.1881651Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:47:32.2167292Z You are using a model of type hunyuan_v1_moe to instantiate a model of type hunyuan. This is not supported for all configurations of models and can yield errors. 2025-10-10T01:47:32.2180857Z INFO 10-10 01:47:32 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:47:38.9621127Z INFO 10-10 01:47:38 [model.py:551] Resolved architecture: HunYuanMoEV1ForCausalLM 2025-10-10T01:47:38.9621749Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:47:38.9866989Z INFO 10-10 01:47:38 [model.py:1545] Using max model len 32768 2025-10-10T01:47:38.9868713Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:47:39.0458016Z INFO 10-10 01:47:39 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:47:39.1683066Z 2025-10-10T01:47:39.1684174Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:47:39.1684552Z tokenizer_config.json: 1.02kB [00:00, 9.23MB/s] 2025-10-10T01:47:39.2610197Z 2025-10-10T01:47:39.6059821Z tokenizer.json: 0% 0.00/16.4M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:40.6922082Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:40.6922481Z 2025-10-10T01:47:40.6922860Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:40.6923215Z 2025-10-10T01:47:40.6923886Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:40.6924504Z 2025-10-10T01:47:40.6924886Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:40.6925266Z 2025-10-10T01:47:40.6925616Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6925977Z 2025-10-10T01:47:40.6926832Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:40.6927409Z 2025-10-10T01:47:40.6927831Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:40.6928218Z 2025-10-10T01:47:40.6928836Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:40.6929335Z 2025-10-10T01:47:40.6929650Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:40.6929953Z 2025-10-10T01:47:40.6930237Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6930545Z 2025-10-10T01:47:40.6931063Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:40.6931742Z 2025-10-10T01:47:40.6932237Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] self._init_executor() 2025-10-10T01:47:40.6932660Z 2025-10-10T01:47:40.6933299Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:40.6934345Z 2025-10-10T01:47:40.6934890Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:40.6935400Z 2025-10-10T01:47:40.6936354Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:40.6936908Z 2025-10-10T01:47:40.6937290Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:40.6937638Z 2025-10-10T01:47:40.6937935Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6938220Z 2025-10-10T01:47:40.6938732Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:40.6939201Z 2025-10-10T01:47:40.6939474Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:40.6939827Z 2025-10-10T01:47:40.6940091Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6940358Z 2025-10-10T01:47:40.6940882Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:40.6941355Z 2025-10-10T01:47:40.6941641Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:40.6941931Z 2025-10-10T01:47:40.6942194Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6942470Z 2025-10-10T01:47:40.6942991Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:40.6943477Z 2025-10-10T01:47:40.6943864Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:40.6944177Z 2025-10-10T01:47:40.6944453Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6944724Z 2025-10-10T01:47:40.6945169Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:40.6945580Z 2025-10-10T01:47:40.6945923Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:40.6946252Z 2025-10-10T01:47:40.6946545Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6946834Z 2025-10-10T01:47:40.6947183Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:40.6947524Z 2025-10-10T01:47:40.6947878Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:40.6948214Z 2025-10-10T01:47:40.6948586Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:40.6948946Z 2025-10-10T01:47:40.6949287Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:40.6949660Z 2025-10-10T01:47:40.6950027Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:40.6950375Z 2025-10-10T01:47:40.6950816Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:40.6951176Z 2025-10-10T01:47:40.6951683Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:40.6952150Z 2025-10-10T01:47:40.6952505Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:40.6952834Z 2025-10-10T01:47:40.6953346Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:40.6953834Z 2025-10-10T01:47:40.6954215Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:40.6954652Z 2025-10-10T01:47:40.6955227Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:40.6955756Z 2025-10-10T01:47:40.6956050Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:40.6956351Z 2025-10-10T01:47:40.6956976Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:40.6957549Z 2025-10-10T01:47:40.6957920Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:40.6958277Z 2025-10-10T01:47:40.6958542Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6958807Z 2025-10-10T01:47:40.6959577Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:40.6960124Z 2025-10-10T01:47:40.6960434Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:40.6960730Z 2025-10-10T01:47:40.6961002Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6961282Z 2025-10-10T01:47:40.6961882Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:40.6962440Z 2025-10-10T01:47:40.6962722Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:40.6963019Z 2025-10-10T01:47:40.6963275Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6963543Z 2025-10-10T01:47:40.6964124Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:40.6964648Z 2025-10-10T01:47:40.6964968Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:40.6965341Z 2025-10-10T01:47:40.6965618Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6965891Z 2025-10-10T01:47:40.6966463Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:40.6966953Z 2025-10-10T01:47:40.6967244Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:40.6967534Z 2025-10-10T01:47:40.6967792Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6968064Z 2025-10-10T01:47:40.6968584Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:40.6969071Z 2025-10-10T01:47:40.6969378Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:40.6969681Z 2025-10-10T01:47:40.6969965Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:40.6970207Z 2025-10-10T01:47:40.6970707Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:40.6971157Z 2025-10-10T01:47:40.6971406Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] raise RuntimeError( 2025-10-10T01:47:40.6971663Z 2025-10-10T01:47:40.6972228Z (EngineCore_DP0 pid=6071) ERROR 10-10 01:47:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:40.6972923Z (EngineCore_DP0 pid=6071) Process EngineCore_DP0: 2025-10-10T01:47:40.6973315Z (EngineCore_DP0 pid=6071) Traceback (most recent call last): 2025-10-10T01:47:40.6973937Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:40.6974503Z (EngineCore_DP0 pid=6071) self.run() 2025-10-10T01:47:40.6975036Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:40.6975597Z (EngineCore_DP0 pid=6071) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:40.6976269Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:40.6976823Z (EngineCore_DP0 pid=6071) raise e 2025-10-10T01:47:40.6977416Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:40.6978046Z (EngineCore_DP0 pid=6071) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:40.6978491Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6979107Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:40.6979764Z (EngineCore_DP0 pid=6071) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:40.6980428Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:40.6981059Z (EngineCore_DP0 pid=6071) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:40.6981562Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6982208Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:40.6982851Z (EngineCore_DP0 pid=6071) self._init_executor() 2025-10-10T01:47:40.6983529Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:40.6984216Z (EngineCore_DP0 pid=6071) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:40.6984949Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:40.6985670Z (EngineCore_DP0 pid=6071) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:40.6986162Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6986811Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:40.6987451Z (EngineCore_DP0 pid=6071) return func(*args, **kwargs) 2025-10-10T01:47:40.6987842Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6988481Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:40.6989107Z (EngineCore_DP0 pid=6071) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:40.6989548Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6990216Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:40.6990875Z (EngineCore_DP0 pid=6071) module = importlib.import_module(module_name) 2025-10-10T01:47:40.6991326Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6991950Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:40.6992582Z (EngineCore_DP0 pid=6071) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:40.6993080Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.6993578Z (EngineCore_DP0 pid=6071) File "", line 1387, in _gcd_import 2025-10-10T01:47:40.6994135Z (EngineCore_DP0 pid=6071) File "", line 1360, in _find_and_load 2025-10-10T01:47:40.6994722Z (EngineCore_DP0 pid=6071) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:40.6995297Z (EngineCore_DP0 pid=6071) File "", line 935, in _load_unlocked 2025-10-10T01:47:40.6995872Z (EngineCore_DP0 pid=6071) File "", line 999, in exec_module 2025-10-10T01:47:40.6996850Z (EngineCore_DP0 pid=6071) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:40.6997598Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:40.6998283Z (EngineCore_DP0 pid=6071) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:40.6999055Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:40.6999884Z (EngineCore_DP0 pid=6071) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:40.7000760Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:40.7001460Z (EngineCore_DP0 pid=6071) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:40.7002253Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:40.7003063Z (EngineCore_DP0 pid=6071) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:40.7003544Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.7004236Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:40.7004931Z (EngineCore_DP0 pid=6071) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:40.7005371Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.7006232Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:40.7006933Z (EngineCore_DP0 pid=6071) return _is_fa2_supported(device)[0] 2025-10-10T01:47:40.7007344Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.7008051Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:40.7008767Z (EngineCore_DP0 pid=6071) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:40.7009212Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.7009884Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:40.7010584Z (EngineCore_DP0 pid=6071) prop = get_device_properties(device) 2025-10-10T01:47:40.7011010Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:40.7011671Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:40.7012333Z (EngineCore_DP0 pid=6071) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:40.7012740Z (EngineCore_DP0 pid=6071) ^^^^^^^^^^^^ 2025-10-10T01:47:40.7013329Z (EngineCore_DP0 pid=6071) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:40.7013884Z (EngineCore_DP0 pid=6071) raise RuntimeError( 2025-10-10T01:47:40.7014576Z (EngineCore_DP0 pid=6071) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:41.1155492Z FAILED 2025-10-10T01:47:41.1286319Z models/test_initialization.py::test_can_initialize_large_subset[Fairseq2LlamaForCausalLM] Fork a new process to run a test 6075 2025-10-10T01:47:41.1297453Z Fork a new process to run a test 0 2025-10-10T01:47:41.1570183Z INFO 10-10 01:47:41 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Fairseq2LlamaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mgleize/fairseq2-dummy-Llama-3.2-1B'} 2025-10-10T01:47:41.2959155Z 2025-10-10T01:47:41.2960733Z config.json: 0% 0.00/591 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:51.9725082Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:51.9725596Z 2025-10-10T01:47:51.9726109Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:51.9726502Z 2025-10-10T01:47:51.9727182Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:51.9727782Z 2025-10-10T01:47:51.9728171Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:51.9728574Z 2025-10-10T01:47:51.9728922Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9729267Z 2025-10-10T01:47:51.9729747Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:51.9730323Z 2025-10-10T01:47:51.9730768Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:51.9731186Z 2025-10-10T01:47:51.9731695Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:51.9732157Z 2025-10-10T01:47:51.9732494Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:51.9732820Z 2025-10-10T01:47:51.9733101Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9733387Z 2025-10-10T01:47:51.9733924Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:51.9734603Z 2025-10-10T01:47:51.9735109Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] self._init_executor() 2025-10-10T01:47:51.9735612Z 2025-10-10T01:47:51.9736433Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:51.9737176Z 2025-10-10T01:47:51.9737540Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:51.9737993Z 2025-10-10T01:47:51.9738741Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:51.9739275Z 2025-10-10T01:47:51.9739636Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:51.9739978Z 2025-10-10T01:47:51.9740268Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9740550Z 2025-10-10T01:47:51.9741054Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:51.9741508Z 2025-10-10T01:47:51.9741787Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:51.9742202Z 2025-10-10T01:47:51.9742467Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9742729Z 2025-10-10T01:47:51.9743291Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:51.9743788Z 2025-10-10T01:47:51.9744077Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:51.9744374Z 2025-10-10T01:47:51.9744639Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9744914Z 2025-10-10T01:47:51.9745437Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:51.9745927Z 2025-10-10T01:47:51.9746229Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:51.9746551Z 2025-10-10T01:47:51.9746870Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9747149Z 2025-10-10T01:47:51.9747609Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:51.9748037Z 2025-10-10T01:47:51.9748387Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:51.9748715Z 2025-10-10T01:47:51.9749015Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9749294Z 2025-10-10T01:47:51.9749640Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:51.9749981Z 2025-10-10T01:47:51.9750331Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:51.9750715Z 2025-10-10T01:47:51.9751103Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:51.9751473Z 2025-10-10T01:47:51.9751815Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:51.9752149Z 2025-10-10T01:47:51.9752517Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:51.9752868Z 2025-10-10T01:47:51.9753246Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:51.9753603Z 2025-10-10T01:47:51.9754116Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:51.9754575Z 2025-10-10T01:47:51.9754931Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:51.9755270Z 2025-10-10T01:47:51.9755806Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:51.9756288Z 2025-10-10T01:47:51.9756664Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:51.9757782Z 2025-10-10T01:47:51.9758392Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:51.9758909Z 2025-10-10T01:47:51.9759355Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:51.9759654Z 2025-10-10T01:47:51.9760281Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:51.9760845Z 2025-10-10T01:47:51.9761212Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:51.9761560Z 2025-10-10T01:47:51.9761817Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9762076Z 2025-10-10T01:47:51.9762638Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:51.9763201Z 2025-10-10T01:47:51.9763505Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:51.9763794Z 2025-10-10T01:47:51.9764070Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9764360Z 2025-10-10T01:47:51.9764969Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:51.9765522Z 2025-10-10T01:47:51.9765804Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:51.9766090Z 2025-10-10T01:47:51.9766349Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9766624Z 2025-10-10T01:47:51.9767266Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:51.9767799Z 2025-10-10T01:47:51.9768127Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:51.9768438Z 2025-10-10T01:47:51.9768719Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9768992Z 2025-10-10T01:47:51.9769517Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:51.9769996Z 2025-10-10T01:47:51.9770276Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:51.9770564Z 2025-10-10T01:47:51.9770822Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9771090Z 2025-10-10T01:47:51.9771607Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:51.9772091Z 2025-10-10T01:47:51.9772398Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:51.9772744Z 2025-10-10T01:47:51.9772978Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:51.9773216Z 2025-10-10T01:47:51.9773737Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:51.9774193Z 2025-10-10T01:47:51.9774445Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] raise RuntimeError( 2025-10-10T01:47:51.9774699Z 2025-10-10T01:47:51.9775261Z (EngineCore_DP0 pid=6154) ERROR 10-10 01:47:51 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:51.9775952Z (EngineCore_DP0 pid=6154) Process EngineCore_DP0: 2025-10-10T01:47:51.9776351Z (EngineCore_DP0 pid=6154) Traceback (most recent call last): 2025-10-10T01:47:51.9776995Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:51.9777513Z (EngineCore_DP0 pid=6154) self.run() 2025-10-10T01:47:51.9778085Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:51.9778652Z (EngineCore_DP0 pid=6154) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:51.9779316Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:51.9779864Z (EngineCore_DP0 pid=6154) raise e 2025-10-10T01:47:51.9780451Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:51.9781085Z (EngineCore_DP0 pid=6154) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:51.9781522Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9782134Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:51.9782827Z (EngineCore_DP0 pid=6154) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:51.9783502Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:51.9784120Z (EngineCore_DP0 pid=6154) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:51.9784582Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9785232Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:51.9785816Z (EngineCore_DP0 pid=6154) self._init_executor() 2025-10-10T01:47:51.9786482Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:51.9787170Z (EngineCore_DP0 pid=6154) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:51.9787891Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:51.9788599Z (EngineCore_DP0 pid=6154) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:51.9789088Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9789726Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:51.9790395Z (EngineCore_DP0 pid=6154) return func(*args, **kwargs) 2025-10-10T01:47:51.9790782Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9791459Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:51.9792081Z (EngineCore_DP0 pid=6154) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:51.9792503Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9793163Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:51.9793814Z (EngineCore_DP0 pid=6154) module = importlib.import_module(module_name) 2025-10-10T01:47:51.9794273Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9794842Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:51.9795541Z (EngineCore_DP0 pid=6154) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:51.9796034Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9796718Z (EngineCore_DP0 pid=6154) File "", line 1387, in _gcd_import 2025-10-10T01:47:51.9797272Z (EngineCore_DP0 pid=6154) File "", line 1360, in _find_and_load 2025-10-10T01:47:51.9797854Z (EngineCore_DP0 pid=6154) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:51.9798437Z (EngineCore_DP0 pid=6154) File "", line 935, in _load_unlocked 2025-10-10T01:47:51.9799005Z (EngineCore_DP0 pid=6154) File "", line 999, in exec_module 2025-10-10T01:47:51.9799663Z (EngineCore_DP0 pid=6154) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:51.9800484Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:51.9801187Z (EngineCore_DP0 pid=6154) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:51.9801906Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:51.9802645Z (EngineCore_DP0 pid=6154) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:51.9803454Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:51.9804134Z (EngineCore_DP0 pid=6154) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:51.9804928Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:51.9805734Z (EngineCore_DP0 pid=6154) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:51.9806211Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9806903Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:51.9807582Z (EngineCore_DP0 pid=6154) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:51.9808111Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9808922Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:51.9809631Z (EngineCore_DP0 pid=6154) return _is_fa2_supported(device)[0] 2025-10-10T01:47:51.9810042Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9810752Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:51.9811463Z (EngineCore_DP0 pid=6154) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:51.9811911Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9812594Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:51.9813232Z (EngineCore_DP0 pid=6154) prop = get_device_properties(device) 2025-10-10T01:47:51.9813731Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:51.9814404Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:51.9815073Z (EngineCore_DP0 pid=6154) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:51.9815465Z (EngineCore_DP0 pid=6154) ^^^^^^^^^^^^ 2025-10-10T01:47:51.9816040Z (EngineCore_DP0 pid=6154) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:51.9816608Z (EngineCore_DP0 pid=6154) raise RuntimeError( 2025-10-10T01:47:51.9817291Z (EngineCore_DP0 pid=6154) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:52.4004820Z FAILED 2025-10-10T01:47:52.4135794Z models/test_initialization.py::test_can_initialize_large_subset[TransformersForCausalLM] Fork a new process to run a test 6158 2025-10-10T01:47:52.4146548Z Fork a new process to run a test 0 2025-10-10T01:47:52.4427125Z INFO 10-10 01:47:52 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='TransformersForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'transformers', 'model': 'hmellor/Ilama-3.2-1B'} 2025-10-10T01:47:53.2168974Z INFO 10-10 01:47:53 [model.py:551] Resolved architecture: TransformersForCausalLM 2025-10-10T01:47:53.2169489Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:47:53.2413531Z INFO 10-10 01:47:53 [model.py:1765] Downcasting torch.float32 to torch.bfloat16. 2025-10-10T01:47:53.2416418Z INFO 10-10 01:47:53 [model.py:1545] Using max model len 131072 2025-10-10T01:47:53.2418859Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:47:53.3007054Z INFO 10-10 01:47:53 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:47:53.9207589Z (EngineCore_DP0 pid=6165) INFO 10-10 01:47:53 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:47:53.9280956Z (EngineCore_DP0 pid=6165) INFO 10-10 01:47:53 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='hmellor/Ilama-3.2-1B', speculative_config=None, tokenizer='hmellor/Ilama-3.2-1B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=hmellor/Ilama-3.2-1B, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:47:54.0644783Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] EngineCore failed to start. 2025-10-10T01:47:54.0645359Z 2025-10-10T01:47:54.0645757Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] Traceback (most recent call last): 2025-10-10T01:47:54.0646161Z 2025-10-10T01:47:54.0646870Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:54.0647495Z 2025-10-10T01:47:54.0648143Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:54.0648552Z 2025-10-10T01:47:54.0648920Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0649263Z 2025-10-10T01:47:54.0650069Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:54.0650779Z 2025-10-10T01:47:54.0651383Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:54.0651736Z 2025-10-10T01:47:54.0652227Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:54.0652699Z 2025-10-10T01:47:54.0653036Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:54.0653349Z 2025-10-10T01:47:54.0653812Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0654249Z 2025-10-10T01:47:54.0655036Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:54.0655971Z 2025-10-10T01:47:54.0656470Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] self._init_executor() 2025-10-10T01:47:54.0656917Z 2025-10-10T01:47:54.0657966Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:54.0658532Z 2025-10-10T01:47:54.0658909Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:54.0659249Z 2025-10-10T01:47:54.0659812Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:54.0660309Z 2025-10-10T01:47:54.0660659Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:54.0661005Z 2025-10-10T01:47:54.0661311Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0661594Z 2025-10-10T01:47:54.0662250Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:54.0662712Z 2025-10-10T01:47:54.0662977Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:47:54.0663247Z 2025-10-10T01:47:54.0663499Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0663755Z 2025-10-10T01:47:54.0664273Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:54.0664751Z 2025-10-10T01:47:54.0665042Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:54.0665333Z 2025-10-10T01:47:54.0665607Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0665880Z 2025-10-10T01:47:54.0666444Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:54.0666948Z 2025-10-10T01:47:54.0667262Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:47:54.0667567Z 2025-10-10T01:47:54.0667839Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0668110Z 2025-10-10T01:47:54.0668561Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:54.0668970Z 2025-10-10T01:47:54.0669321Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:54.0669647Z 2025-10-10T01:47:54.0669938Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0670212Z 2025-10-10T01:47:54.0670549Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:47:54.0670888Z 2025-10-10T01:47:54.0671234Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:47:54.0671632Z 2025-10-10T01:47:54.0672020Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:54.0672387Z 2025-10-10T01:47:54.0672778Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:47:54.0673128Z 2025-10-10T01:47:54.0673505Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:47:54.0673855Z 2025-10-10T01:47:54.0674233Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:54.0674588Z 2025-10-10T01:47:54.0675093Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:54.0675554Z 2025-10-10T01:47:54.0675908Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:54.0676248Z 2025-10-10T01:47:54.0676827Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:54.0677315Z 2025-10-10T01:47:54.0677687Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:54.0678062Z 2025-10-10T01:47:54.0678621Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:54.0679269Z 2025-10-10T01:47:54.0679567Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:47:54.0679869Z 2025-10-10T01:47:54.0680517Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:54.0681110Z 2025-10-10T01:47:54.0681531Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:54.0681885Z 2025-10-10T01:47:54.0682160Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0682422Z 2025-10-10T01:47:54.0682991Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:54.0683506Z 2025-10-10T01:47:54.0683809Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:47:54.0684106Z 2025-10-10T01:47:54.0684397Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0684678Z 2025-10-10T01:47:54.0685277Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:54.0685824Z 2025-10-10T01:47:54.0686105Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:47:54.0686385Z 2025-10-10T01:47:54.0686648Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0687019Z 2025-10-10T01:47:54.0687612Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:54.0688136Z 2025-10-10T01:47:54.0688508Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:54.0688823Z 2025-10-10T01:47:54.0689100Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0689388Z 2025-10-10T01:47:54.0689910Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:54.0690416Z 2025-10-10T01:47:54.0690745Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:47:54.0691089Z 2025-10-10T01:47:54.0691395Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0691715Z 2025-10-10T01:47:54.0692334Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:54.0692973Z 2025-10-10T01:47:54.0693293Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:47:54.0693604Z 2025-10-10T01:47:54.0693835Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:47:54.0694074Z 2025-10-10T01:47:54.0694548Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:54.0695002Z 2025-10-10T01:47:54.0695252Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] raise RuntimeError( 2025-10-10T01:47:54.0695517Z 2025-10-10T01:47:54.0696292Z (EngineCore_DP0 pid=6165) ERROR 10-10 01:47:54 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:54.0697105Z (EngineCore_DP0 pid=6165) Process EngineCore_DP0: 2025-10-10T01:47:54.0697532Z (EngineCore_DP0 pid=6165) Traceback (most recent call last): 2025-10-10T01:47:54.0698157Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:47:54.0698681Z (EngineCore_DP0 pid=6165) self.run() 2025-10-10T01:47:54.0699206Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:47:54.0699771Z (EngineCore_DP0 pid=6165) self._target(*self._args, **self._kwargs) 2025-10-10T01:47:54.0700469Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:47:54.0701124Z (EngineCore_DP0 pid=6165) raise e 2025-10-10T01:47:54.0701817Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:47:54.0702496Z (EngineCore_DP0 pid=6165) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:47:54.0702931Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0703560Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:47:54.0704281Z (EngineCore_DP0 pid=6165) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:47:54.0704942Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:47:54.0705622Z (EngineCore_DP0 pid=6165) self.model_executor = executor_class(vllm_config) 2025-10-10T01:47:54.0706096Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0706747Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:47:54.0707342Z (EngineCore_DP0 pid=6165) self._init_executor() 2025-10-10T01:47:54.0707999Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:47:54.0708677Z (EngineCore_DP0 pid=6165) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:47:54.0709399Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:47:54.0710201Z (EngineCore_DP0 pid=6165) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:47:54.0710712Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0711360Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:47:54.0711948Z (EngineCore_DP0 pid=6165) return func(*args, **kwargs) 2025-10-10T01:47:54.0712340Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0712977Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:47:54.0713607Z (EngineCore_DP0 pid=6165) worker_class = resolve_obj_by_qualname( 2025-10-10T01:47:54.0714028Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0714742Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:47:54.0715417Z (EngineCore_DP0 pid=6165) module = importlib.import_module(module_name) 2025-10-10T01:47:54.0715865Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0716454Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:47:54.0717079Z (EngineCore_DP0 pid=6165) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:47:54.0717572Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0718065Z (EngineCore_DP0 pid=6165) File "", line 1387, in _gcd_import 2025-10-10T01:47:54.0718622Z (EngineCore_DP0 pid=6165) File "", line 1360, in _find_and_load 2025-10-10T01:47:54.0719305Z (EngineCore_DP0 pid=6165) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:47:54.0719885Z (EngineCore_DP0 pid=6165) File "", line 935, in _load_unlocked 2025-10-10T01:47:54.0720446Z (EngineCore_DP0 pid=6165) File "", line 999, in exec_module 2025-10-10T01:47:54.0721047Z (EngineCore_DP0 pid=6165) File "", line 488, in _call_with_frames_removed 2025-10-10T01:47:54.0721871Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:47:54.0722547Z (EngineCore_DP0 pid=6165) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:47:54.0723326Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:47:54.0724062Z (EngineCore_DP0 pid=6165) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:47:54.0724859Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:47:54.0725529Z (EngineCore_DP0 pid=6165) class FlashAttentionMetadataBuilder( 2025-10-10T01:47:54.0726314Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:47:54.0727132Z (EngineCore_DP0 pid=6165) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:47:54.0727608Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0728363Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:47:54.0729058Z (EngineCore_DP0 pid=6165) if not is_fa_version_supported(fa_version): 2025-10-10T01:47:54.0729493Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0730242Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:47:54.0730939Z (EngineCore_DP0 pid=6165) return _is_fa2_supported(device)[0] 2025-10-10T01:47:54.0731340Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0732050Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:47:54.0732807Z (EngineCore_DP0 pid=6165) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:47:54.0733265Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0733943Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:47:54.0734574Z (EngineCore_DP0 pid=6165) prop = get_device_properties(device) 2025-10-10T01:47:54.0734979Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:47:54.0735633Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:47:54.0736292Z (EngineCore_DP0 pid=6165) _lazy_init() # will define _get_device_properties 2025-10-10T01:47:54.0736709Z (EngineCore_DP0 pid=6165) ^^^^^^^^^^^^ 2025-10-10T01:47:54.0737291Z (EngineCore_DP0 pid=6165) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:47:54.0737848Z (EngineCore_DP0 pid=6165) raise RuntimeError( 2025-10-10T01:47:54.0738532Z (EngineCore_DP0 pid=6165) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:47:54.4817805Z FAILED 2025-10-10T01:47:54.4954281Z models/test_initialization.py::test_can_initialize_large_subset[SkyworkR1VChatModel] Fork a new process to run a test 6169 2025-10-10T01:47:54.4965907Z Fork a new process to run a test 0 2025-10-10T01:47:54.5242954Z INFO 10-10 01:47:54 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='SkyworkR1VChatModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Skywork/Skywork-R1V-38B'} 2025-10-10T01:47:54.6005709Z 2025-10-10T01:47:54.6007198Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:47:54.6007527Z config.json: 3.71kB [00:00, 19.8MB/s] 2025-10-10T01:47:54.6961392Z 2025-10-10T01:47:54.6962964Z configuration_skywork_chat.py: 0.00B [00:00, ?B/s] 2025-10-10T01:47:54.6963377Z configuration_skywork_chat.py: 3.77kB [00:00, 16.5MB/s] 2025-10-10T01:47:54.7373237Z 2025-10-10T01:47:54.7375337Z configuration_skywork_vit.py: 0.00B [00:00, ?B/s] 2025-10-10T01:47:54.7375776Z configuration_skywork_vit.py: 4.52kB [00:00, 20.0MB/s] 2025-10-10T01:47:54.7395159Z A new version of the following files was downloaded from https://huggingface.co/Skywork/Skywork-R1V-38B: 2025-10-10T01:47:54.7395699Z - configuration_skywork_vit.py 2025-10-10T01:47:54.7396559Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:47:54.7761365Z 2025-10-10T01:47:54.7764394Z configuration_skywork_lm2.py: 0.00B [00:00, ?B/s] 2025-10-10T01:47:54.7764723Z configuration_skywork_lm2.py: 6.40kB [00:00, 19.5MB/s] 2025-10-10T01:47:54.7785891Z A new version of the following files was downloaded from https://huggingface.co/Skywork/Skywork-R1V-38B: 2025-10-10T01:47:54.7786409Z - configuration_skywork_lm2.py 2025-10-10T01:47:54.7787025Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:47:54.7787875Z A new version of the following files was downloaded from https://huggingface.co/Skywork/Skywork-R1V-38B: 2025-10-10T01:47:54.7788382Z - configuration_skywork_chat.py 2025-10-10T01:47:54.7788646Z - configuration_skywork_vit.py 2025-10-10T01:47:54.7788907Z - configuration_skywork_lm2.py 2025-10-10T01:47:54.7789514Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:47:55.0313186Z 2025-10-10T01:47:55.0313627Z preprocessor_config.json: 0% 0.00/287 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:48:03.0535869Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] EngineCore failed to start. 2025-10-10T01:48:03.0536583Z 2025-10-10T01:48:03.0537133Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] Traceback (most recent call last): 2025-10-10T01:48:03.0537519Z 2025-10-10T01:48:03.0538206Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:03.0538798Z 2025-10-10T01:48:03.0539231Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:03.0539577Z 2025-10-10T01:48:03.0539870Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0540169Z 2025-10-10T01:48:03.0540681Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:03.0541145Z 2025-10-10T01:48:03.0541609Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:03.0542178Z 2025-10-10T01:48:03.0542910Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:03.0543519Z 2025-10-10T01:48:03.0543846Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:03.0544152Z 2025-10-10T01:48:03.0544434Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0544836Z 2025-10-10T01:48:03.0545729Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:03.0546464Z 2025-10-10T01:48:03.0546784Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] self._init_executor() 2025-10-10T01:48:03.0547058Z 2025-10-10T01:48:03.0548105Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:03.0548916Z 2025-10-10T01:48:03.0549285Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:03.0549615Z 2025-10-10T01:48:03.0550172Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:03.0550833Z 2025-10-10T01:48:03.0551190Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:03.0551529Z 2025-10-10T01:48:03.0551815Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0552094Z 2025-10-10T01:48:03.0552587Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:03.0553061Z 2025-10-10T01:48:03.0553333Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:48:03.0553601Z 2025-10-10T01:48:03.0553857Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0554116Z 2025-10-10T01:48:03.0554704Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:03.0555210Z 2025-10-10T01:48:03.0555503Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:03.0555800Z 2025-10-10T01:48:03.0556065Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0556346Z 2025-10-10T01:48:03.0556884Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:03.0557373Z 2025-10-10T01:48:03.0557680Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:48:03.0557983Z 2025-10-10T01:48:03.0558264Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0558538Z 2025-10-10T01:48:03.0558987Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:03.0559525Z 2025-10-10T01:48:03.0559878Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:03.0560264Z 2025-10-10T01:48:03.0560552Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0560833Z 2025-10-10T01:48:03.0561210Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:48:03.0561548Z 2025-10-10T01:48:03.0561904Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:48:03.0562252Z 2025-10-10T01:48:03.0562624Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:03.0562979Z 2025-10-10T01:48:03.0563326Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:48:03.0563656Z 2025-10-10T01:48:03.0564018Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:48:03.0564450Z 2025-10-10T01:48:03.0564842Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:03.0565249Z 2025-10-10T01:48:03.0565761Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:03.0566230Z 2025-10-10T01:48:03.0566583Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:03.0566924Z 2025-10-10T01:48:03.0567464Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:03.0567953Z 2025-10-10T01:48:03.0568323Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:03.0568679Z 2025-10-10T01:48:03.0569290Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:03.0577409Z 2025-10-10T01:48:03.0577814Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:48:03.0578138Z 2025-10-10T01:48:03.0578816Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:03.0579400Z 2025-10-10T01:48:03.0579804Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:03.0580164Z 2025-10-10T01:48:03.0580433Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0580706Z 2025-10-10T01:48:03.0581305Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:03.0581835Z 2025-10-10T01:48:03.0582143Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:48:03.0582453Z 2025-10-10T01:48:03.0582728Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0583006Z 2025-10-10T01:48:03.0583613Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:03.0584266Z 2025-10-10T01:48:03.0584605Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:48:03.0584912Z 2025-10-10T01:48:03.0585183Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0585453Z 2025-10-10T01:48:03.0586038Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:03.0586562Z 2025-10-10T01:48:03.0586887Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:03.0587208Z 2025-10-10T01:48:03.0587492Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0587769Z 2025-10-10T01:48:03.0588291Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:03.0588838Z 2025-10-10T01:48:03.0589130Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:48:03.0589423Z 2025-10-10T01:48:03.0589684Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0589958Z 2025-10-10T01:48:03.0590473Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:03.0590977Z 2025-10-10T01:48:03.0591291Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:48:03.0591595Z 2025-10-10T01:48:03.0591823Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:48:03.0592071Z 2025-10-10T01:48:03.0592612Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:03.0593066Z 2025-10-10T01:48:03.0593321Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] raise RuntimeError( 2025-10-10T01:48:03.0593579Z 2025-10-10T01:48:03.0594144Z (EngineCore_DP0 pid=6227) ERROR 10-10 01:48:03 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:03.0594847Z (EngineCore_DP0 pid=6227) Process EngineCore_DP0: 2025-10-10T01:48:03.0595270Z (EngineCore_DP0 pid=6227) Traceback (most recent call last): 2025-10-10T01:48:03.0595881Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:48:03.0596664Z (EngineCore_DP0 pid=6227) self.run() 2025-10-10T01:48:03.0597214Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:48:03.0597790Z (EngineCore_DP0 pid=6227) self._target(*self._args, **self._kwargs) 2025-10-10T01:48:03.0598458Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:48:03.0599084Z (EngineCore_DP0 pid=6227) raise e 2025-10-10T01:48:03.0599680Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:03.0600446Z (EngineCore_DP0 pid=6227) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:03.0600959Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0601598Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:03.0602239Z (EngineCore_DP0 pid=6227) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:03.0602915Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:03.0603534Z (EngineCore_DP0 pid=6227) self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:03.0603992Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0604641Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:03.0605226Z (EngineCore_DP0 pid=6227) self._init_executor() 2025-10-10T01:48:03.0605990Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:03.0606677Z (EngineCore_DP0 pid=6227) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:03.0607404Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:03.0608108Z (EngineCore_DP0 pid=6227) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:03.0608631Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0609400Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:03.0610108Z (EngineCore_DP0 pid=6227) return func(*args, **kwargs) 2025-10-10T01:48:03.0610571Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0611377Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:03.0612023Z (EngineCore_DP0 pid=6227) worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:03.0612451Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0613125Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:03.0613783Z (EngineCore_DP0 pid=6227) module = importlib.import_module(module_name) 2025-10-10T01:48:03.0614231Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0614819Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:03.0615463Z (EngineCore_DP0 pid=6227) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:03.0615956Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0616455Z (EngineCore_DP0 pid=6227) File "", line 1387, in _gcd_import 2025-10-10T01:48:03.0617018Z (EngineCore_DP0 pid=6227) File "", line 1360, in _find_and_load 2025-10-10T01:48:03.0617606Z (EngineCore_DP0 pid=6227) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:03.0618238Z (EngineCore_DP0 pid=6227) File "", line 935, in _load_unlocked 2025-10-10T01:48:03.0618905Z (EngineCore_DP0 pid=6227) File "", line 999, in exec_module 2025-10-10T01:48:03.0619629Z (EngineCore_DP0 pid=6227) File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:03.0620496Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:03.0621176Z (EngineCore_DP0 pid=6227) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:03.0621903Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:03.0622631Z (EngineCore_DP0 pid=6227) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:03.0623420Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:03.0624143Z (EngineCore_DP0 pid=6227) class FlashAttentionMetadataBuilder( 2025-10-10T01:48:03.0624952Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:03.0625782Z (EngineCore_DP0 pid=6227) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:03.0626269Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0626965Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:03.0627654Z (EngineCore_DP0 pid=6227) if not is_fa_version_supported(fa_version): 2025-10-10T01:48:03.0628094Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0628885Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:03.0629609Z (EngineCore_DP0 pid=6227) return _is_fa2_supported(device)[0] 2025-10-10T01:48:03.0630025Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0630737Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:03.0631450Z (EngineCore_DP0 pid=6227) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:03.0631903Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0632570Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:03.0633213Z (EngineCore_DP0 pid=6227) prop = get_device_properties(device) 2025-10-10T01:48:03.0633635Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:03.0634304Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:03.0634959Z (EngineCore_DP0 pid=6227) _lazy_init() # will define _get_device_properties 2025-10-10T01:48:03.0635357Z (EngineCore_DP0 pid=6227) ^^^^^^^^^^^^ 2025-10-10T01:48:03.0635936Z (EngineCore_DP0 pid=6227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:03.0636549Z (EngineCore_DP0 pid=6227) raise RuntimeError( 2025-10-10T01:48:03.0637286Z (EngineCore_DP0 pid=6227) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:03.4979570Z FAILED 2025-10-10T01:48:03.5110145Z models/test_initialization.py::test_can_initialize_large_subset[InternLM2ForCausalLM] Fork a new process to run a test 6231 2025-10-10T01:48:03.5120320Z Fork a new process to run a test 0 2025-10-10T01:48:03.5397524Z INFO 10-10 01:48:03 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternLM2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'internlm/internlm2-chat-7b'} 2025-10-10T01:48:03.6524912Z 2025-10-10T01:48:03.6525231Z config.json: 0% 0.00/916 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:48:12.4621486Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] EngineCore failed to start. 2025-10-10T01:48:12.4622131Z 2025-10-10T01:48:12.4622559Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] Traceback (most recent call last): 2025-10-10T01:48:12.4622954Z 2025-10-10T01:48:12.4623632Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:12.4624266Z 2025-10-10T01:48:12.4624676Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:12.4625054Z 2025-10-10T01:48:12.4625398Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4625737Z 2025-10-10T01:48:12.4626420Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:12.4627448Z 2025-10-10T01:48:12.4628152Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:12.4628898Z 2025-10-10T01:48:12.4629836Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:12.4630682Z 2025-10-10T01:48:12.4631400Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:12.4631968Z 2025-10-10T01:48:12.4632469Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4632964Z 2025-10-10T01:48:12.4633849Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:12.4634370Z 2025-10-10T01:48:12.4634631Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] self._init_executor() 2025-10-10T01:48:12.4634898Z 2025-10-10T01:48:12.4635444Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:12.4636060Z 2025-10-10T01:48:12.4636403Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:12.4636723Z 2025-10-10T01:48:12.4637260Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:12.4637758Z 2025-10-10T01:48:12.4638098Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:12.4638430Z 2025-10-10T01:48:12.4638715Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4638994Z 2025-10-10T01:48:12.4639618Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:12.4640085Z 2025-10-10T01:48:12.4640439Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:48:12.4640742Z 2025-10-10T01:48:12.4641002Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4641258Z 2025-10-10T01:48:12.4641773Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:12.4642243Z 2025-10-10T01:48:12.4642536Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:12.4642818Z 2025-10-10T01:48:12.4643089Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4643357Z 2025-10-10T01:48:12.4643887Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:12.4644382Z 2025-10-10T01:48:12.4644726Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:48:12.4645024Z 2025-10-10T01:48:12.4645302Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4645572Z 2025-10-10T01:48:12.4646018Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:12.4646484Z 2025-10-10T01:48:12.4646822Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:12.4647201Z 2025-10-10T01:48:12.4647493Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4647779Z 2025-10-10T01:48:12.4648126Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:48:12.4648459Z 2025-10-10T01:48:12.4648804Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:48:12.4649141Z 2025-10-10T01:48:12.4649519Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:12.4649877Z 2025-10-10T01:48:12.4650224Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:48:12.4650604Z 2025-10-10T01:48:12.4650985Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:48:12.4651336Z 2025-10-10T01:48:12.4651713Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:12.4652071Z 2025-10-10T01:48:12.4652579Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:12.4653042Z 2025-10-10T01:48:12.4653387Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:12.4653721Z 2025-10-10T01:48:12.4654234Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:12.4654716Z 2025-10-10T01:48:12.4655144Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:12.4655507Z 2025-10-10T01:48:12.4656052Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:12.4656559Z 2025-10-10T01:48:12.4656857Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:48:12.4657152Z 2025-10-10T01:48:12.4657774Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:12.4658340Z 2025-10-10T01:48:12.4658710Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:12.4659054Z 2025-10-10T01:48:12.4659306Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4659558Z 2025-10-10T01:48:12.4660123Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:12.4660637Z 2025-10-10T01:48:12.4660930Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:48:12.4661283Z 2025-10-10T01:48:12.4661559Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4661829Z 2025-10-10T01:48:12.4662470Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:12.4663024Z 2025-10-10T01:48:12.4663310Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:48:12.4663589Z 2025-10-10T01:48:12.4663849Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4664108Z 2025-10-10T01:48:12.4664701Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:12.4665251Z 2025-10-10T01:48:12.4665583Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:12.4665961Z 2025-10-10T01:48:12.4666246Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4666528Z 2025-10-10T01:48:12.4667063Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:12.4667555Z 2025-10-10T01:48:12.4667840Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:48:12.4668128Z 2025-10-10T01:48:12.4668386Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4668654Z 2025-10-10T01:48:12.4669166Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:12.4669643Z 2025-10-10T01:48:12.4669999Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:48:12.4670304Z 2025-10-10T01:48:12.4670532Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:48:12.4670770Z 2025-10-10T01:48:12.4671262Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:12.4671725Z 2025-10-10T01:48:12.4671970Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] raise RuntimeError( 2025-10-10T01:48:12.4672230Z 2025-10-10T01:48:12.4672794Z (EngineCore_DP0 pid=6310) ERROR 10-10 01:48:12 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:12.4673489Z (EngineCore_DP0 pid=6310) Process EngineCore_DP0: 2025-10-10T01:48:12.4673887Z (EngineCore_DP0 pid=6310) Traceback (most recent call last): 2025-10-10T01:48:12.4674490Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:48:12.4675007Z (EngineCore_DP0 pid=6310) self.run() 2025-10-10T01:48:12.4675524Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:48:12.4676082Z (EngineCore_DP0 pid=6310) self._target(*self._args, **self._kwargs) 2025-10-10T01:48:12.4676804Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:48:12.4677350Z (EngineCore_DP0 pid=6310) raise e 2025-10-10T01:48:12.4677985Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:12.4678620Z (EngineCore_DP0 pid=6310) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:12.4679128Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4679777Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:12.4680414Z (EngineCore_DP0 pid=6310) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:12.4681070Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:12.4681675Z (EngineCore_DP0 pid=6310) self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:12.4682125Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4682869Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:12.4683449Z (EngineCore_DP0 pid=6310) self._init_executor() 2025-10-10T01:48:12.4684111Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:12.4684791Z (EngineCore_DP0 pid=6310) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:12.4685513Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:12.4686232Z (EngineCore_DP0 pid=6310) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:12.4686734Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4687423Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:12.4688028Z (EngineCore_DP0 pid=6310) return func(*args, **kwargs) 2025-10-10T01:48:12.4688425Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4689058Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:12.4689700Z (EngineCore_DP0 pid=6310) worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:12.4690129Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4690788Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:12.4691439Z (EngineCore_DP0 pid=6310) module = importlib.import_module(module_name) 2025-10-10T01:48:12.4691897Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4692471Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:12.4693092Z (EngineCore_DP0 pid=6310) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:12.4693574Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4694129Z (EngineCore_DP0 pid=6310) File "", line 1387, in _gcd_import 2025-10-10T01:48:12.4694681Z (EngineCore_DP0 pid=6310) File "", line 1360, in _find_and_load 2025-10-10T01:48:12.4695309Z (EngineCore_DP0 pid=6310) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:12.4695891Z (EngineCore_DP0 pid=6310) File "", line 935, in _load_unlocked 2025-10-10T01:48:12.4697140Z (EngineCore_DP0 pid=6310) File "", line 999, in exec_module 2025-10-10T01:48:12.4697784Z (EngineCore_DP0 pid=6310) File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:12.4698545Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:12.4699246Z (EngineCore_DP0 pid=6310) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:12.4699991Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:12.4700875Z (EngineCore_DP0 pid=6310) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:12.4701671Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:12.4702343Z (EngineCore_DP0 pid=6310) class FlashAttentionMetadataBuilder( 2025-10-10T01:48:12.4703135Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:12.4703946Z (EngineCore_DP0 pid=6310) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:12.4704424Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4705114Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:12.4705860Z (EngineCore_DP0 pid=6310) if not is_fa_version_supported(fa_version): 2025-10-10T01:48:12.4706324Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4707083Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:12.4707789Z (EngineCore_DP0 pid=6310) return _is_fa2_supported(device)[0] 2025-10-10T01:48:12.4708203Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4708919Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:12.4709629Z (EngineCore_DP0 pid=6310) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:12.4710079Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4710753Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:12.4711390Z (EngineCore_DP0 pid=6310) prop = get_device_properties(device) 2025-10-10T01:48:12.4711806Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:12.4712463Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:12.4713193Z (EngineCore_DP0 pid=6310) _lazy_init() # will define _get_device_properties 2025-10-10T01:48:12.4713590Z (EngineCore_DP0 pid=6310) ^^^^^^^^^^^^ 2025-10-10T01:48:12.4714234Z (EngineCore_DP0 pid=6310) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:12.4714796Z (EngineCore_DP0 pid=6310) raise RuntimeError( 2025-10-10T01:48:12.4715475Z (EngineCore_DP0 pid=6310) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:12.8760742Z FAILED 2025-10-10T01:48:12.8891250Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5OmniForConditionalGeneration] Fork a new process to run a test 6314 2025-10-10T01:48:12.8902016Z Fork a new process to run a test 0 2025-10-10T01:48:12.9179380Z INFO 10-10 01:48:12 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2_5OmniForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen2.5-Omni-7B-AWQ'} 2025-10-10T01:48:13.1171287Z 2025-10-10T01:48:13.1174032Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:13.1174330Z config.json: 15.4kB [00:00, 66.5MB/s] 2025-10-10T01:48:13.1540832Z Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'} 2025-10-10T01:48:13.2300799Z 2025-10-10T01:48:13.2302290Z preprocessor_config.json: 0% 0.00/667 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:48:15.2263037Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] EngineCore failed to start. 2025-10-10T01:48:15.2263485Z 2025-10-10T01:48:15.2264152Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] Traceback (most recent call last): 2025-10-10T01:48:15.2264527Z 2025-10-10T01:48:15.2265206Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:15.2265794Z 2025-10-10T01:48:15.2266185Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:15.2266571Z 2025-10-10T01:48:15.2266913Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2267262Z 2025-10-10T01:48:15.2267883Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:15.2268607Z 2025-10-10T01:48:15.2269112Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:15.2269538Z 2025-10-10T01:48:15.2270147Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:15.2270713Z 2025-10-10T01:48:15.2271024Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:15.2271458Z 2025-10-10T01:48:15.2271747Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2272022Z 2025-10-10T01:48:15.2272850Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:15.2273662Z 2025-10-10T01:48:15.2274134Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] self._init_executor() 2025-10-10T01:48:15.2274600Z 2025-10-10T01:48:15.2275536Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:15.2276317Z 2025-10-10T01:48:15.2276710Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:15.2277053Z 2025-10-10T01:48:15.2277643Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:15.2278155Z 2025-10-10T01:48:15.2278669Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:15.2279130Z 2025-10-10T01:48:15.2279465Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2279761Z 2025-10-10T01:48:15.2280276Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:15.2280738Z 2025-10-10T01:48:15.2281004Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:48:15.2281279Z 2025-10-10T01:48:15.2281526Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2281786Z 2025-10-10T01:48:15.2282302Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:15.2282863Z 2025-10-10T01:48:15.2283160Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:15.2283446Z 2025-10-10T01:48:15.2283714Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2283981Z 2025-10-10T01:48:15.2284508Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:15.2284988Z 2025-10-10T01:48:15.2285292Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:48:15.2285586Z 2025-10-10T01:48:15.2285857Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2286147Z 2025-10-10T01:48:15.2286585Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:15.2286995Z 2025-10-10T01:48:15.2287329Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:15.2287659Z 2025-10-10T01:48:15.2287939Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2288262Z 2025-10-10T01:48:15.2288606Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:48:15.2288931Z 2025-10-10T01:48:15.2289328Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:48:15.2289668Z 2025-10-10T01:48:15.2290044Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:15.2290427Z 2025-10-10T01:48:15.2290785Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:48:15.2291121Z 2025-10-10T01:48:15.2291487Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:48:15.2291837Z 2025-10-10T01:48:15.2292214Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:15.2292704Z 2025-10-10T01:48:15.2293220Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:15.2293753Z 2025-10-10T01:48:15.2294107Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:15.2294453Z 2025-10-10T01:48:15.2294966Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:15.2295446Z 2025-10-10T01:48:15.2295817Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:15.2296396Z 2025-10-10T01:48:15.2296957Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:15.2297466Z 2025-10-10T01:48:15.2297851Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:48:15.2298162Z 2025-10-10T01:48:15.2298801Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:15.2299371Z 2025-10-10T01:48:15.2299741Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:15.2300086Z 2025-10-10T01:48:15.2300351Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2300607Z 2025-10-10T01:48:15.2301161Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:15.2301693Z 2025-10-10T01:48:15.2301988Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:48:15.2302284Z 2025-10-10T01:48:15.2302552Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2302825Z 2025-10-10T01:48:15.2303416Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:15.2304076Z 2025-10-10T01:48:15.2304360Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:48:15.2304641Z 2025-10-10T01:48:15.2304959Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2305242Z 2025-10-10T01:48:15.2305836Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:15.2306372Z 2025-10-10T01:48:15.2306697Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:15.2307014Z 2025-10-10T01:48:15.2307295Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2307575Z 2025-10-10T01:48:15.2308105Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:15.2308590Z 2025-10-10T01:48:15.2308868Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:48:15.2309240Z 2025-10-10T01:48:15.2309508Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2309780Z 2025-10-10T01:48:15.2310295Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:15.2310769Z 2025-10-10T01:48:15.2311080Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:48:15.2311381Z 2025-10-10T01:48:15.2311617Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:48:15.2311854Z 2025-10-10T01:48:15.2312342Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:15.2312790Z 2025-10-10T01:48:15.2313081Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] raise RuntimeError( 2025-10-10T01:48:15.2313347Z 2025-10-10T01:48:15.2313917Z (EngineCore_DP0 pid=6343) ERROR 10-10 01:48:15 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:15.2314611Z (EngineCore_DP0 pid=6343) Process EngineCore_DP0: 2025-10-10T01:48:15.2315013Z (EngineCore_DP0 pid=6343) Traceback (most recent call last): 2025-10-10T01:48:15.2315622Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:48:15.2316132Z (EngineCore_DP0 pid=6343) self.run() 2025-10-10T01:48:15.2316658Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:48:15.2317239Z (EngineCore_DP0 pid=6343) self._target(*self._args, **self._kwargs) 2025-10-10T01:48:15.2317913Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:48:15.2318468Z (EngineCore_DP0 pid=6343) raise e 2025-10-10T01:48:15.2319129Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:15.2319789Z (EngineCore_DP0 pid=6343) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:15.2320289Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2320965Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:15.2321627Z (EngineCore_DP0 pid=6343) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:15.2322285Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:15.2322896Z (EngineCore_DP0 pid=6343) self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:15.2323347Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2324006Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:15.2324590Z (EngineCore_DP0 pid=6343) self._init_executor() 2025-10-10T01:48:15.2325253Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:15.2326004Z (EngineCore_DP0 pid=6343) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:15.2326747Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:15.2327459Z (EngineCore_DP0 pid=6343) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:15.2327955Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2328596Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:15.2329179Z (EngineCore_DP0 pid=6343) return func(*args, **kwargs) 2025-10-10T01:48:15.2329567Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2330262Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:15.2330894Z (EngineCore_DP0 pid=6343) worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:15.2331315Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2331969Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:15.2332623Z (EngineCore_DP0 pid=6343) module = importlib.import_module(module_name) 2025-10-10T01:48:15.2333067Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2333644Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:15.2334266Z (EngineCore_DP0 pid=6343) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:15.2334770Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2335264Z (EngineCore_DP0 pid=6343) File "", line 1387, in _gcd_import 2025-10-10T01:48:15.2335812Z (EngineCore_DP0 pid=6343) File "", line 1360, in _find_and_load 2025-10-10T01:48:15.2336396Z (EngineCore_DP0 pid=6343) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:15.2336969Z (EngineCore_DP0 pid=6343) File "", line 935, in _load_unlocked 2025-10-10T01:48:15.2337581Z (EngineCore_DP0 pid=6343) File "", line 999, in exec_module 2025-10-10T01:48:15.2338204Z (EngineCore_DP0 pid=6343) File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:15.2338937Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:15.2339616Z (EngineCore_DP0 pid=6343) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:15.2340335Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:15.2341053Z (EngineCore_DP0 pid=6343) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:15.2341840Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:15.2342507Z (EngineCore_DP0 pid=6343) class FlashAttentionMetadataBuilder( 2025-10-10T01:48:15.2343298Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:15.2344168Z (EngineCore_DP0 pid=6343) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:15.2344646Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2345334Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:15.2346008Z (EngineCore_DP0 pid=6343) if not is_fa_version_supported(fa_version): 2025-10-10T01:48:15.2346457Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2347206Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:15.2347901Z (EngineCore_DP0 pid=6343) return _is_fa2_supported(device)[0] 2025-10-10T01:48:15.2348355Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2349061Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:15.2349771Z (EngineCore_DP0 pid=6343) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:15.2350224Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2350898Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:15.2351542Z (EngineCore_DP0 pid=6343) prop = get_device_properties(device) 2025-10-10T01:48:15.2351981Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:15.2352667Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:15.2353338Z (EngineCore_DP0 pid=6343) _lazy_init() # will define _get_device_properties 2025-10-10T01:48:15.2353736Z (EngineCore_DP0 pid=6343) ^^^^^^^^^^^^ 2025-10-10T01:48:15.2354322Z (EngineCore_DP0 pid=6343) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:15.2354879Z (EngineCore_DP0 pid=6343) raise RuntimeError( 2025-10-10T01:48:15.2355609Z (EngineCore_DP0 pid=6343) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:15.6538888Z FAILED 2025-10-10T01:48:15.6670228Z models/test_initialization.py::test_can_initialize_large_subset[ChatGLMForConditionalGeneration] Fork a new process to run a test 6347 2025-10-10T01:48:15.6680869Z Fork a new process to run a test 0 2025-10-10T01:48:15.6958810Z INFO 10-10 01:48:15 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ChatGLMForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'thu-coai/ShieldLM-6B-chatglm3'} 2025-10-10T01:48:15.8329976Z 2025-10-10T01:48:15.8332310Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:15.8332796Z config.json: 1.51kB [00:00, 7.54MB/s] 2025-10-10T01:48:23.0209936Z INFO 10-10 01:48:23 [model.py:551] Resolved architecture: ChatGLMForConditionalGeneration 2025-10-10T01:48:23.0210807Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:48:23.0453191Z INFO 10-10 01:48:23 [model.py:1545] Using max model len 8192 2025-10-10T01:48:23.0455693Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:48:23.2365660Z INFO 10-10 01:48:23 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:48:23.2887198Z 2025-10-10T01:48:23.2888338Z tokenizer_config.json: 0% 0.00/244 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:48:24.1245855Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] EngineCore failed to start. 2025-10-10T01:48:24.1246410Z 2025-10-10T01:48:24.1246852Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] Traceback (most recent call last): 2025-10-10T01:48:24.1247354Z 2025-10-10T01:48:24.1248160Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:24.1248777Z 2025-10-10T01:48:24.1249168Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:24.1249556Z 2025-10-10T01:48:24.1249898Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1250247Z 2025-10-10T01:48:24.1250863Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:24.1251672Z 2025-10-10T01:48:24.1252121Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:24.1252658Z 2025-10-10T01:48:24.1253355Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:24.1253936Z 2025-10-10T01:48:24.1254328Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:24.1254720Z 2025-10-10T01:48:24.1255065Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1255414Z 2025-10-10T01:48:24.1256045Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:24.1256763Z 2025-10-10T01:48:24.1257218Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] self._init_executor() 2025-10-10T01:48:24.1257562Z 2025-10-10T01:48:24.1258553Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:24.1259344Z 2025-10-10T01:48:24.1260006Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:24.1260754Z 2025-10-10T01:48:24.1261392Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:24.1262033Z 2025-10-10T01:48:24.1262412Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:24.1262774Z 2025-10-10T01:48:24.1263082Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1263369Z 2025-10-10T01:48:24.1263878Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:24.1264347Z 2025-10-10T01:48:24.1264617Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:48:24.1264892Z 2025-10-10T01:48:24.1265142Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1265495Z 2025-10-10T01:48:24.1266020Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:24.1266510Z 2025-10-10T01:48:24.1266820Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:24.1267116Z 2025-10-10T01:48:24.1267389Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1267660Z 2025-10-10T01:48:24.1268212Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:24.1268712Z 2025-10-10T01:48:24.1269017Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:48:24.1269329Z 2025-10-10T01:48:24.1269650Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1269933Z 2025-10-10T01:48:24.1270380Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:24.1270808Z 2025-10-10T01:48:24.1271152Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:24.1271488Z 2025-10-10T01:48:24.1271771Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1272053Z 2025-10-10T01:48:24.1272398Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:48:24.1272728Z 2025-10-10T01:48:24.1273086Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:48:24.1273431Z 2025-10-10T01:48:24.1273807Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:24.1274163Z 2025-10-10T01:48:24.1274500Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:48:24.1274834Z 2025-10-10T01:48:24.1275192Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:48:24.1275605Z 2025-10-10T01:48:24.1275977Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:24.1276346Z 2025-10-10T01:48:24.1276884Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:24.1277350Z 2025-10-10T01:48:24.1277700Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:24.1278034Z 2025-10-10T01:48:24.1278564Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:24.1279188Z 2025-10-10T01:48:24.1279572Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:24.1279951Z 2025-10-10T01:48:24.1280510Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:24.1281085Z 2025-10-10T01:48:24.1281392Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:48:24.1281694Z 2025-10-10T01:48:24.1282323Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:24.1282897Z 2025-10-10T01:48:24.1283266Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:24.1283618Z 2025-10-10T01:48:24.1283869Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1284139Z 2025-10-10T01:48:24.1284738Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:24.1285264Z 2025-10-10T01:48:24.1285561Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:48:24.1285859Z 2025-10-10T01:48:24.1286126Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1286397Z 2025-10-10T01:48:24.1287001Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:24.1287553Z 2025-10-10T01:48:24.1287842Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:48:24.1288126Z 2025-10-10T01:48:24.1288394Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1288663Z 2025-10-10T01:48:24.1289258Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:24.1289784Z 2025-10-10T01:48:24.1290099Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:24.1290410Z 2025-10-10T01:48:24.1290679Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1291003Z 2025-10-10T01:48:24.1291523Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:24.1292062Z 2025-10-10T01:48:24.1292360Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:48:24.1292656Z 2025-10-10T01:48:24.1292918Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1293185Z 2025-10-10T01:48:24.1293721Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:24.1294203Z 2025-10-10T01:48:24.1294511Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:48:24.1294816Z 2025-10-10T01:48:24.1295044Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:48:24.1295287Z 2025-10-10T01:48:24.1295773Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:24.1296668Z 2025-10-10T01:48:24.1296942Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] raise RuntimeError( 2025-10-10T01:48:24.1297215Z 2025-10-10T01:48:24.1297782Z (EngineCore_DP0 pid=6425) ERROR 10-10 01:48:24 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:24.1298503Z (EngineCore_DP0 pid=6425) Process EngineCore_DP0: 2025-10-10T01:48:24.1298903Z (EngineCore_DP0 pid=6425) Traceback (most recent call last): 2025-10-10T01:48:24.1299510Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:48:24.1300023Z (EngineCore_DP0 pid=6425) self.run() 2025-10-10T01:48:24.1300652Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:48:24.1301237Z (EngineCore_DP0 pid=6425) self._target(*self._args, **self._kwargs) 2025-10-10T01:48:24.1301902Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:48:24.1302452Z (EngineCore_DP0 pid=6425) raise e 2025-10-10T01:48:24.1303042Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:24.1303670Z (EngineCore_DP0 pid=6425) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:24.1304111Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1304731Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:24.1305372Z (EngineCore_DP0 pid=6425) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:24.1306035Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:24.1306652Z (EngineCore_DP0 pid=6425) self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:24.1307099Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1307860Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:24.1308444Z (EngineCore_DP0 pid=6425) self._init_executor() 2025-10-10T01:48:24.1309168Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:24.1309893Z (EngineCore_DP0 pid=6425) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:24.1310624Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:24.1311336Z (EngineCore_DP0 pid=6425) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:24.1311847Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1312503Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:24.1313097Z (EngineCore_DP0 pid=6425) return func(*args, **kwargs) 2025-10-10T01:48:24.1313479Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1314195Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:24.1314827Z (EngineCore_DP0 pid=6425) worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:24.1315253Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1315917Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:24.1316565Z (EngineCore_DP0 pid=6425) module = importlib.import_module(module_name) 2025-10-10T01:48:24.1317011Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1317597Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:24.1318279Z (EngineCore_DP0 pid=6425) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:24.1318777Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1319360Z (EngineCore_DP0 pid=6425) File "", line 1387, in _gcd_import 2025-10-10T01:48:24.1319914Z (EngineCore_DP0 pid=6425) File "", line 1360, in _find_and_load 2025-10-10T01:48:24.1320502Z (EngineCore_DP0 pid=6425) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:24.1321083Z (EngineCore_DP0 pid=6425) File "", line 935, in _load_unlocked 2025-10-10T01:48:24.1321648Z (EngineCore_DP0 pid=6425) File "", line 999, in exec_module 2025-10-10T01:48:24.1322244Z (EngineCore_DP0 pid=6425) File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:24.1322976Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:24.1323657Z (EngineCore_DP0 pid=6425) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:24.1324382Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:24.1325101Z (EngineCore_DP0 pid=6425) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:24.1325948Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:24.1326650Z (EngineCore_DP0 pid=6425) class FlashAttentionMetadataBuilder( 2025-10-10T01:48:24.1327448Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:24.1328278Z (EngineCore_DP0 pid=6425) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:24.1328755Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1329452Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:24.1330132Z (EngineCore_DP0 pid=6425) if not is_fa_version_supported(fa_version): 2025-10-10T01:48:24.1330568Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1331315Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:24.1332074Z (EngineCore_DP0 pid=6425) return _is_fa2_supported(device)[0] 2025-10-10T01:48:24.1332486Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1333200Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:24.1333908Z (EngineCore_DP0 pid=6425) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:24.1334361Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1335030Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:24.1335671Z (EngineCore_DP0 pid=6425) prop = get_device_properties(device) 2025-10-10T01:48:24.1336097Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:24.1336806Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:24.1337489Z (EngineCore_DP0 pid=6425) _lazy_init() # will define _get_device_properties 2025-10-10T01:48:24.1337885Z (EngineCore_DP0 pid=6425) ^^^^^^^^^^^^ 2025-10-10T01:48:24.1338462Z (EngineCore_DP0 pid=6425) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:24.1339036Z (EngineCore_DP0 pid=6425) raise RuntimeError( 2025-10-10T01:48:24.1339721Z (EngineCore_DP0 pid=6425) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:24.5326418Z FAILED 2025-10-10T01:48:24.5457519Z models/test_initialization.py::test_can_initialize_large_subset[EagleDeepSeekMTPModel] Fork a new process to run a test 6429 2025-10-10T01:48:24.5469218Z Fork a new process to run a test 0 2025-10-10T01:48:24.5747009Z INFO 10-10 01:48:24 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='EagleDeepSeekMTPModel', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'eagle618/eagle-deepseek-v3-random', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'eagle618/deepseek-v3-random'} 2025-10-10T01:48:24.6890460Z 2025-10-10T01:48:24.6893088Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:24.6893384Z config.json: 1.95kB [00:00, 7.07MB/s] 2025-10-10T01:48:24.7316474Z INFO 10-10 01:48:24 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:48:24.7887878Z INFO 10-10 01:48:24 [model.py:551] Resolved architecture: DeepseekV3ForCausalLM 2025-10-10T01:48:24.7888378Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:48:24.8137481Z INFO 10-10 01:48:24 [model.py:1545] Using max model len 163840 2025-10-10T01:48:24.9145018Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:48:25.2137800Z 2025-10-10T01:48:25.2138583Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:25.2138964Z config.json: 1.79kB [00:00, 13.0MB/s] 2025-10-10T01:48:25.2590573Z INFO 10-10 01:48:25 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:48:25.3220012Z INFO 10-10 01:48:25 [model.py:551] Resolved architecture: DeepSeekMTPModel 2025-10-10T01:48:25.3222145Z INFO 10-10 01:48:25 [model.py:1545] Using max model len 163840 2025-10-10T01:48:25.3230974Z INFO 10-10 01:48:25 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:48:25.3632071Z 2025-10-10T01:48:25.3633718Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:25.3634089Z tokenizer_config.json: 3.13kB [00:00, 17.1MB/s] 2025-10-10T01:48:25.4906592Z 2025-10-10T01:48:25.5193478Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:25.5193821Z tokenizer.json: 7.85MB [00:00, 273MB/s] 2025-10-10T01:48:26.0746526Z INFO 10-10 01:48:26 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:48:26.0949178Z (EngineCore_DP0 pid=6437) INFO 10-10 01:48:26 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:48:26.1024674Z (EngineCore_DP0 pid=6437) INFO 10-10 01:48:26 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='eagle618/deepseek-v3-random', speculative_config=SpeculativeConfig(method='eagle', model='eagle618/eagle-deepseek-v3-random', num_spec_tokens=1), tokenizer='eagle618/deepseek-v3-random', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=163840, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=eagle618/deepseek-v3-random, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': ['+quant_fp8'], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:48:26.2343982Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] EngineCore failed to start. 2025-10-10T01:48:26.2344541Z 2025-10-10T01:48:26.2345417Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] Traceback (most recent call last): 2025-10-10T01:48:26.2345810Z 2025-10-10T01:48:26.2346508Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:26.2347096Z 2025-10-10T01:48:26.2347486Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:26.2347862Z 2025-10-10T01:48:26.2348199Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2348553Z 2025-10-10T01:48:26.2349153Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:26.2349828Z 2025-10-10T01:48:26.2350252Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:26.2350627Z 2025-10-10T01:48:26.2351252Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:26.2351693Z 2025-10-10T01:48:26.2352034Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:26.2352341Z 2025-10-10T01:48:26.2352623Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2352899Z 2025-10-10T01:48:26.2353425Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:26.2353901Z 2025-10-10T01:48:26.2354258Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] self._init_executor() 2025-10-10T01:48:26.2354526Z 2025-10-10T01:48:26.2355063Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:26.2355817Z 2025-10-10T01:48:26.2356435Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:26.2357004Z 2025-10-10T01:48:26.2357883Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:26.2358810Z 2025-10-10T01:48:26.2359593Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:26.2360121Z 2025-10-10T01:48:26.2360445Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2360728Z 2025-10-10T01:48:26.2361220Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:26.2361665Z 2025-10-10T01:48:26.2361937Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:48:26.2362365Z 2025-10-10T01:48:26.2362631Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2362891Z 2025-10-10T01:48:26.2363462Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:26.2363962Z 2025-10-10T01:48:26.2364252Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:26.2364549Z 2025-10-10T01:48:26.2364815Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2365086Z 2025-10-10T01:48:26.2365619Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:26.2366109Z 2025-10-10T01:48:26.2366413Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:48:26.2366710Z 2025-10-10T01:48:26.2366986Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2367306Z 2025-10-10T01:48:26.2367761Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:26.2368179Z 2025-10-10T01:48:26.2368532Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:26.2368862Z 2025-10-10T01:48:26.2369143Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2369424Z 2025-10-10T01:48:26.2369768Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:48:26.2370103Z 2025-10-10T01:48:26.2370445Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:48:26.2370787Z 2025-10-10T01:48:26.2371207Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:26.2371574Z 2025-10-10T01:48:26.2371925Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:48:26.2372251Z 2025-10-10T01:48:26.2372614Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:48:26.2372959Z 2025-10-10T01:48:26.2373335Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:26.2373689Z 2025-10-10T01:48:26.2374192Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:26.2374653Z 2025-10-10T01:48:26.2375010Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:26.2375365Z 2025-10-10T01:48:26.2375892Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:26.2376375Z 2025-10-10T01:48:26.2376745Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:26.2377150Z 2025-10-10T01:48:26.2377700Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:26.2378220Z 2025-10-10T01:48:26.2378556Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:48:26.2378857Z 2025-10-10T01:48:26.2379502Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:26.2380072Z 2025-10-10T01:48:26.2380448Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:26.2380801Z 2025-10-10T01:48:26.2381060Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2381324Z 2025-10-10T01:48:26.2381889Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:26.2382457Z 2025-10-10T01:48:26.2382754Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:48:26.2383051Z 2025-10-10T01:48:26.2383320Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2383596Z 2025-10-10T01:48:26.2384192Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:26.2384745Z 2025-10-10T01:48:26.2385023Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:48:26.2385304Z 2025-10-10T01:48:26.2385565Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2385831Z 2025-10-10T01:48:26.2386477Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:26.2387000Z 2025-10-10T01:48:26.2387323Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:26.2387641Z 2025-10-10T01:48:26.2387917Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2388190Z 2025-10-10T01:48:26.2388708Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:26.2389195Z 2025-10-10T01:48:26.2396460Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:48:26.2396919Z 2025-10-10T01:48:26.2397267Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2397557Z 2025-10-10T01:48:26.2398108Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:26.2398605Z 2025-10-10T01:48:26.2398924Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:48:26.2399339Z 2025-10-10T01:48:26.2399713Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:48:26.2399972Z 2025-10-10T01:48:26.2400466Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:26.2400998Z 2025-10-10T01:48:26.2401259Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] raise RuntimeError( 2025-10-10T01:48:26.2401530Z 2025-10-10T01:48:26.2402109Z (EngineCore_DP0 pid=6437) ERROR 10-10 01:48:26 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:26.2402828Z (EngineCore_DP0 pid=6437) Process EngineCore_DP0: 2025-10-10T01:48:26.2403229Z (EngineCore_DP0 pid=6437) Traceback (most recent call last): 2025-10-10T01:48:26.2403830Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:48:26.2404355Z (EngineCore_DP0 pid=6437) self.run() 2025-10-10T01:48:26.2404888Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:48:26.2405554Z (EngineCore_DP0 pid=6437) self._target(*self._args, **self._kwargs) 2025-10-10T01:48:26.2406239Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:48:26.2406782Z (EngineCore_DP0 pid=6437) raise e 2025-10-10T01:48:26.2407365Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:26.2407999Z (EngineCore_DP0 pid=6437) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:26.2408447Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2409067Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:26.2409706Z (EngineCore_DP0 pid=6437) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:26.2410438Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:26.2411081Z (EngineCore_DP0 pid=6437) self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:26.2411537Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2412196Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:26.2412824Z (EngineCore_DP0 pid=6437) self._init_executor() 2025-10-10T01:48:26.2413514Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:26.2414222Z (EngineCore_DP0 pid=6437) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:26.2414958Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:26.2415681Z (EngineCore_DP0 pid=6437) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:26.2416192Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2416850Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:26.2417508Z (EngineCore_DP0 pid=6437) return func(*args, **kwargs) 2025-10-10T01:48:26.2417894Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2418582Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:26.2419234Z (EngineCore_DP0 pid=6437) worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:26.2419663Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2420339Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:26.2420997Z (EngineCore_DP0 pid=6437) module = importlib.import_module(module_name) 2025-10-10T01:48:26.2421439Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2422036Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:26.2422655Z (EngineCore_DP0 pid=6437) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:26.2423213Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2423708Z (EngineCore_DP0 pid=6437) File "", line 1387, in _gcd_import 2025-10-10T01:48:26.2424262Z (EngineCore_DP0 pid=6437) File "", line 1360, in _find_and_load 2025-10-10T01:48:26.2424853Z (EngineCore_DP0 pid=6437) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:26.2425424Z (EngineCore_DP0 pid=6437) File "", line 935, in _load_unlocked 2025-10-10T01:48:26.2425999Z (EngineCore_DP0 pid=6437) File "", line 999, in exec_module 2025-10-10T01:48:26.2426604Z (EngineCore_DP0 pid=6437) File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:26.2427351Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:26.2428131Z (EngineCore_DP0 pid=6437) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:26.2428867Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:26.2429607Z (EngineCore_DP0 pid=6437) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:26.2430417Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:26.2431105Z (EngineCore_DP0 pid=6437) class FlashAttentionMetadataBuilder( 2025-10-10T01:48:26.2431908Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:26.2432732Z (EngineCore_DP0 pid=6437) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:26.2433208Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2433907Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:26.2434588Z (EngineCore_DP0 pid=6437) if not is_fa_version_supported(fa_version): 2025-10-10T01:48:26.2435026Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2435821Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:26.2436551Z (EngineCore_DP0 pid=6437) return _is_fa2_supported(device)[0] 2025-10-10T01:48:26.2436979Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2437708Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:26.2438422Z (EngineCore_DP0 pid=6437) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:26.2438874Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2439641Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:26.2440279Z (EngineCore_DP0 pid=6437) prop = get_device_properties(device) 2025-10-10T01:48:26.2440697Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:26.2441414Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:26.2442076Z (EngineCore_DP0 pid=6437) _lazy_init() # will define _get_device_properties 2025-10-10T01:48:26.2442473Z (EngineCore_DP0 pid=6437) ^^^^^^^^^^^^ 2025-10-10T01:48:26.2443047Z (EngineCore_DP0 pid=6437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:26.2443628Z (EngineCore_DP0 pid=6437) raise RuntimeError( 2025-10-10T01:48:26.2444312Z (EngineCore_DP0 pid=6437) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:26.6527742Z FAILED 2025-10-10T01:48:26.6657490Z models/test_initialization.py::test_can_initialize_large_subset[PersimmonForCausalLM] Fork a new process to run a test 6441 2025-10-10T01:48:26.6668927Z Fork a new process to run a test 0 2025-10-10T01:48:26.6945738Z INFO 10-10 01:48:26 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='PersimmonForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'adept/persimmon-8b-chat'} 2025-10-10T01:48:26.7794070Z 2025-10-10T01:48:26.7795108Z config.json: 0% 0.00/635 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message 2025-10-10T01:48:35.2167850Z You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. 2025-10-10T01:48:57.5056637Z 2025-10-10T01:48:57.5057126Z generation_config.json: 0% 0.00/124 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:48:57.6610474Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] EngineCore failed to start. 2025-10-10T01:48:57.6610940Z 2025-10-10T01:48:57.6611357Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] Traceback (most recent call last): 2025-10-10T01:48:57.6612347Z 2025-10-10T01:48:57.6613466Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:57.6614085Z 2025-10-10T01:48:57.6614633Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:57.6615037Z 2025-10-10T01:48:57.6615399Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6615748Z 2025-10-10T01:48:57.6616352Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:57.6616914Z 2025-10-10T01:48:57.6617560Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:57.6618003Z 2025-10-10T01:48:57.6618661Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:57.6619284Z 2025-10-10T01:48:57.6619651Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:57.6620140Z 2025-10-10T01:48:57.6620462Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6620734Z 2025-10-10T01:48:57.6621582Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:57.6622182Z 2025-10-10T01:48:57.6622667Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] self._init_executor() 2025-10-10T01:48:57.6623064Z 2025-10-10T01:48:57.6623788Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:57.6624582Z 2025-10-10T01:48:57.6625048Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:57.6625528Z 2025-10-10T01:48:57.6626101Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:57.6626608Z 2025-10-10T01:48:57.6626953Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:57.6627292Z 2025-10-10T01:48:57.6627580Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6627860Z 2025-10-10T01:48:57.6628349Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:57.6628798Z 2025-10-10T01:48:57.6629074Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:48:57.6629349Z 2025-10-10T01:48:57.6629600Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6629855Z 2025-10-10T01:48:57.6630376Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:57.6630845Z 2025-10-10T01:48:57.6631145Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:57.6631500Z 2025-10-10T01:48:57.6631769Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6632056Z 2025-10-10T01:48:57.6632652Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:57.6633157Z 2025-10-10T01:48:57.6633465Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:48:57.6633774Z 2025-10-10T01:48:57.6634054Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6634329Z 2025-10-10T01:48:57.6634783Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:57.6635194Z 2025-10-10T01:48:57.6635539Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:57.6635868Z 2025-10-10T01:48:57.6636203Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6636487Z 2025-10-10T01:48:57.6636828Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:48:57.6637160Z 2025-10-10T01:48:57.6637505Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:48:57.6637857Z 2025-10-10T01:48:57.6638287Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:57.6638649Z 2025-10-10T01:48:57.6639139Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:48:57.6639480Z 2025-10-10T01:48:57.6639847Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:48:57.6640253Z 2025-10-10T01:48:57.6640639Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:57.6640997Z 2025-10-10T01:48:57.6641499Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:57.6641959Z 2025-10-10T01:48:57.6642315Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:57.6642648Z 2025-10-10T01:48:57.6643163Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:57.6643649Z 2025-10-10T01:48:57.6644019Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:57.6644381Z 2025-10-10T01:48:57.6645073Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:57.6645876Z 2025-10-10T01:48:57.6646333Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:48:57.6646805Z 2025-10-10T01:48:57.6647943Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:57.6648991Z 2025-10-10T01:48:57.6649727Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:57.6650300Z 2025-10-10T01:48:57.6650719Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6651145Z 2025-10-10T01:48:57.6651889Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:57.6652412Z 2025-10-10T01:48:57.6652712Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:48:57.6653019Z 2025-10-10T01:48:57.6653294Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6653568Z 2025-10-10T01:48:57.6654169Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:57.6654834Z 2025-10-10T01:48:57.6655118Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:48:57.6655404Z 2025-10-10T01:48:57.6655666Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6655930Z 2025-10-10T01:48:57.6656512Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:57.6657037Z 2025-10-10T01:48:57.6657374Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:57.6657685Z 2025-10-10T01:48:57.6657963Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6658276Z 2025-10-10T01:48:57.6658797Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:57.6659288Z 2025-10-10T01:48:57.6659567Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:48:57.6659855Z 2025-10-10T01:48:57.6660114Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6660387Z 2025-10-10T01:48:57.6660899Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:57.6661377Z 2025-10-10T01:48:57.6661686Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:48:57.6661991Z 2025-10-10T01:48:57.6662220Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:48:57.6662459Z 2025-10-10T01:48:57.6662945Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:57.6663392Z 2025-10-10T01:48:57.6663639Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] raise RuntimeError( 2025-10-10T01:48:57.6663949Z 2025-10-10T01:48:57.6664513Z (EngineCore_DP0 pid=6519) ERROR 10-10 01:48:57 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:57.6665240Z (EngineCore_DP0 pid=6519) Process EngineCore_DP0: 2025-10-10T01:48:57.6665643Z (EngineCore_DP0 pid=6519) Traceback (most recent call last): 2025-10-10T01:48:57.6666273Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:48:57.6666798Z (EngineCore_DP0 pid=6519) self.run() 2025-10-10T01:48:57.6667325Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:48:57.6667893Z (EngineCore_DP0 pid=6519) self._target(*self._args, **self._kwargs) 2025-10-10T01:48:57.6668549Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:48:57.6669105Z (EngineCore_DP0 pid=6519) raise e 2025-10-10T01:48:57.6669692Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:48:57.6670373Z (EngineCore_DP0 pid=6519) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:48:57.6670811Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6671437Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:48:57.6672083Z (EngineCore_DP0 pid=6519) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:48:57.6672740Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:48:57.6673362Z (EngineCore_DP0 pid=6519) self.model_executor = executor_class(vllm_config) 2025-10-10T01:48:57.6673818Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6674511Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:48:57.6675099Z (EngineCore_DP0 pid=6519) self._init_executor() 2025-10-10T01:48:57.6675762Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:48:57.6676444Z (EngineCore_DP0 pid=6519) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:48:57.6677185Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:48:57.6677894Z (EngineCore_DP0 pid=6519) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:48:57.6678392Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6679147Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:48:57.6679753Z (EngineCore_DP0 pid=6519) return func(*args, **kwargs) 2025-10-10T01:48:57.6680132Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6680763Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:48:57.6681384Z (EngineCore_DP0 pid=6519) worker_class = resolve_obj_by_qualname( 2025-10-10T01:48:57.6681872Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6682541Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:48:57.6683231Z (EngineCore_DP0 pid=6519) module = importlib.import_module(module_name) 2025-10-10T01:48:57.6683677Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6684249Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:48:57.6684871Z (EngineCore_DP0 pid=6519) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:48:57.6685360Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6685856Z (EngineCore_DP0 pid=6519) File "", line 1387, in _gcd_import 2025-10-10T01:48:57.6686410Z (EngineCore_DP0 pid=6519) File "", line 1360, in _find_and_load 2025-10-10T01:48:57.6686990Z (EngineCore_DP0 pid=6519) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:48:57.6687610Z (EngineCore_DP0 pid=6519) File "", line 935, in _load_unlocked 2025-10-10T01:48:57.6688179Z (EngineCore_DP0 pid=6519) File "", line 999, in exec_module 2025-10-10T01:48:57.6688855Z (EngineCore_DP0 pid=6519) File "", line 488, in _call_with_frames_removed 2025-10-10T01:48:57.6689720Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:48:57.6690528Z (EngineCore_DP0 pid=6519) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:48:57.6691326Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:48:57.6692046Z (EngineCore_DP0 pid=6519) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:48:57.6692876Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:48:57.6693573Z (EngineCore_DP0 pid=6519) class FlashAttentionMetadataBuilder( 2025-10-10T01:48:57.6694372Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:48:57.6695183Z (EngineCore_DP0 pid=6519) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:48:57.6695677Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6696592Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:48:57.6697284Z (EngineCore_DP0 pid=6519) if not is_fa_version_supported(fa_version): 2025-10-10T01:48:57.6697722Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6698469Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:48:57.6699298Z (EngineCore_DP0 pid=6519) return _is_fa2_supported(device)[0] 2025-10-10T01:48:57.6699780Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6700713Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:48:57.6701428Z (EngineCore_DP0 pid=6519) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:48:57.6701954Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6702642Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:48:57.6703284Z (EngineCore_DP0 pid=6519) prop = get_device_properties(device) 2025-10-10T01:48:57.6703705Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:48:57.6704372Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:48:57.6705038Z (EngineCore_DP0 pid=6519) _lazy_init() # will define _get_device_properties 2025-10-10T01:48:57.6705450Z (EngineCore_DP0 pid=6519) ^^^^^^^^^^^^ 2025-10-10T01:48:57.6706039Z (EngineCore_DP0 pid=6519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:48:57.6706680Z (EngineCore_DP0 pid=6519) raise RuntimeError( 2025-10-10T01:48:57.6707368Z (EngineCore_DP0 pid=6519) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:48:58.1211186Z FAILED 2025-10-10T01:48:58.1342650Z models/test_initialization.py::test_can_initialize_large_subset[Eagle3Qwen2_5vlForCausalLM] Fork a new process to run a test 6523 2025-10-10T01:48:58.1354013Z Fork a new process to run a test 0 2025-10-10T01:48:58.1631689Z INFO 10-10 01:48:58 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Eagle3Qwen2_5vlForCausalLM', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'Rayzl/qwen2.5-vl-7b-eagle3-sgl', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'Qwen/Qwen2.5-VL-7B-Instruct'} 2025-10-10T01:48:58.2582281Z 2025-10-10T01:48:58.2584422Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:48:58.2584895Z config.json: 1.37kB [00:00, 8.53MB/s] 2025-10-10T01:48:58.3943736Z 2025-10-10T01:48:58.3944519Z preprocessor_config.json: 0% 0.00/350 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:49:13.2340658Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] EngineCore failed to start. 2025-10-10T01:49:13.2341252Z 2025-10-10T01:49:13.2341661Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] Traceback (most recent call last): 2025-10-10T01:49:13.2342078Z 2025-10-10T01:49:13.2343328Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:13.2343988Z 2025-10-10T01:49:13.2344322Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:13.2344625Z 2025-10-10T01:49:13.2344898Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2345179Z 2025-10-10T01:49:13.2345838Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:13.2346812Z 2025-10-10T01:49:13.2347267Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:13.2347604Z 2025-10-10T01:49:13.2348114Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:13.2348579Z 2025-10-10T01:49:13.2348901Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:13.2349228Z 2025-10-10T01:49:13.2349756Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2350243Z 2025-10-10T01:49:13.2351158Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:13.2351973Z 2025-10-10T01:49:13.2352420Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] self._init_executor() 2025-10-10T01:49:13.2353023Z 2025-10-10T01:49:13.2353847Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:13.2354358Z 2025-10-10T01:49:13.2354692Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:13.2355018Z 2025-10-10T01:49:13.2355559Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:13.2356093Z 2025-10-10T01:49:13.2356462Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:13.2356809Z 2025-10-10T01:49:13.2357104Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2357391Z 2025-10-10T01:49:13.2358003Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:13.2358466Z 2025-10-10T01:49:13.2358743Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:49:13.2359022Z 2025-10-10T01:49:13.2359403Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2359666Z 2025-10-10T01:49:13.2360186Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:13.2360655Z 2025-10-10T01:49:13.2360947Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:13.2361242Z 2025-10-10T01:49:13.2361509Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2361783Z 2025-10-10T01:49:13.2362304Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:13.2362793Z 2025-10-10T01:49:13.2363102Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:49:13.2363465Z 2025-10-10T01:49:13.2363749Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2364021Z 2025-10-10T01:49:13.2364509Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:13.2364924Z 2025-10-10T01:49:13.2365283Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:13.2365614Z 2025-10-10T01:49:13.2365903Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2366185Z 2025-10-10T01:49:13.2366525Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:49:13.2366872Z 2025-10-10T01:49:13.2367223Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:49:13.2367565Z 2025-10-10T01:49:13.2367940Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:13.2368987Z 2025-10-10T01:49:13.2369349Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:49:13.2369687Z 2025-10-10T01:49:13.2370055Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:49:13.2370406Z 2025-10-10T01:49:13.2370787Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:13.2371147Z 2025-10-10T01:49:13.2371652Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:13.2372119Z 2025-10-10T01:49:13.2372478Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:13.2372872Z 2025-10-10T01:49:13.2373412Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:13.2373896Z 2025-10-10T01:49:13.2374266Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:13.2374632Z 2025-10-10T01:49:13.2375178Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:13.2375688Z 2025-10-10T01:49:13.2375977Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:49:13.2376273Z 2025-10-10T01:49:13.2376903Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:13.2377465Z 2025-10-10T01:49:13.2377832Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:13.2378179Z 2025-10-10T01:49:13.2378434Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2378690Z 2025-10-10T01:49:13.2379321Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:13.2379829Z 2025-10-10T01:49:13.2380163Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:49:13.2380464Z 2025-10-10T01:49:13.2380738Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2381015Z 2025-10-10T01:49:13.2381609Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:13.2382155Z 2025-10-10T01:49:13.2382430Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:49:13.2382716Z 2025-10-10T01:49:13.2382971Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2383230Z 2025-10-10T01:49:13.2383810Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:13.2384377Z 2025-10-10T01:49:13.2384702Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:13.2385007Z 2025-10-10T01:49:13.2385280Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2385551Z 2025-10-10T01:49:13.2386104Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:13.2386581Z 2025-10-10T01:49:13.2386862Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:49:13.2387145Z 2025-10-10T01:49:13.2387410Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2387697Z 2025-10-10T01:49:13.2388261Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:13.2388749Z 2025-10-10T01:49:13.2389056Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:49:13.2389362Z 2025-10-10T01:49:13.2389585Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:49:13.2389821Z 2025-10-10T01:49:13.2390306Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:13.2390751Z 2025-10-10T01:49:13.2391000Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] raise RuntimeError( 2025-10-10T01:49:13.2391258Z 2025-10-10T01:49:13.2391833Z (EngineCore_DP0 pid=6631) ERROR 10-10 01:49:13 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:13.2392513Z (EngineCore_DP0 pid=6631) Process EngineCore_DP0: 2025-10-10T01:49:13.2392909Z (EngineCore_DP0 pid=6631) Traceback (most recent call last): 2025-10-10T01:49:13.2393515Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:49:13.2394039Z (EngineCore_DP0 pid=6631) self.run() 2025-10-10T01:49:13.2394625Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:49:13.2395180Z (EngineCore_DP0 pid=6631) self._target(*self._args, **self._kwargs) 2025-10-10T01:49:13.2395889Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:49:13.2396765Z (EngineCore_DP0 pid=6631) raise e 2025-10-10T01:49:13.2397383Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:13.2398024Z (EngineCore_DP0 pid=6631) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:13.2398458Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2399139Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:13.2399785Z (EngineCore_DP0 pid=6631) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:13.2400453Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:13.2401203Z (EngineCore_DP0 pid=6631) self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:13.2401694Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2402367Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:13.2402960Z (EngineCore_DP0 pid=6631) self._init_executor() 2025-10-10T01:49:13.2403629Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:13.2404316Z (EngineCore_DP0 pid=6631) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:13.2405036Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:13.2405823Z (EngineCore_DP0 pid=6631) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:13.2406324Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2406965Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:13.2407551Z (EngineCore_DP0 pid=6631) return func(*args, **kwargs) 2025-10-10T01:49:13.2407938Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2408566Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:13.2409198Z (EngineCore_DP0 pid=6631) worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:13.2409629Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2410294Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:13.2410951Z (EngineCore_DP0 pid=6631) module = importlib.import_module(module_name) 2025-10-10T01:49:13.2411387Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2411966Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:13.2412666Z (EngineCore_DP0 pid=6631) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:13.2413160Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2413732Z (EngineCore_DP0 pid=6631) File "", line 1387, in _gcd_import 2025-10-10T01:49:13.2414307Z (EngineCore_DP0 pid=6631) File "", line 1360, in _find_and_load 2025-10-10T01:49:13.2414897Z (EngineCore_DP0 pid=6631) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:13.2415475Z (EngineCore_DP0 pid=6631) File "", line 935, in _load_unlocked 2025-10-10T01:49:13.2416045Z (EngineCore_DP0 pid=6631) File "", line 999, in exec_module 2025-10-10T01:49:13.2416648Z (EngineCore_DP0 pid=6631) File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:13.2417370Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:13.2418114Z (EngineCore_DP0 pid=6631) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:13.2418849Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:13.2419573Z (EngineCore_DP0 pid=6631) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:13.2420356Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:13.2421039Z (EngineCore_DP0 pid=6631) class FlashAttentionMetadataBuilder( 2025-10-10T01:49:13.2421822Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:13.2422637Z (EngineCore_DP0 pid=6631) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:13.2423160Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2423865Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:13.2424547Z (EngineCore_DP0 pid=6631) if not is_fa_version_supported(fa_version): 2025-10-10T01:49:13.2424981Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2425718Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:13.2426413Z (EngineCore_DP0 pid=6631) return _is_fa2_supported(device)[0] 2025-10-10T01:49:13.2426821Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2427552Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:13.2428260Z (EngineCore_DP0 pid=6631) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:13.2428712Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2429378Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:13.2430072Z (EngineCore_DP0 pid=6631) prop = get_device_properties(device) 2025-10-10T01:49:13.2430490Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:13.2431182Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:13.2431852Z (EngineCore_DP0 pid=6631) _lazy_init() # will define _get_device_properties 2025-10-10T01:49:13.2432244Z (EngineCore_DP0 pid=6631) ^^^^^^^^^^^^ 2025-10-10T01:49:13.2432836Z (EngineCore_DP0 pid=6631) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:13.2433408Z (EngineCore_DP0 pid=6631) raise RuntimeError( 2025-10-10T01:49:13.2434083Z (EngineCore_DP0 pid=6631) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:13.6512997Z FAILED 2025-10-10T01:49:13.6644333Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2AudioForConditionalGeneration] Fork a new process to run a test 6635 2025-10-10T01:49:13.6655249Z Fork a new process to run a test 0 2025-10-10T01:49:13.6934904Z INFO 10-10 01:49:13 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2AudioForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen2-Audio-7B-Instruct'} 2025-10-10T01:49:13.8045011Z 2025-10-10T01:49:13.8046159Z config.json: 0% 0.00/853 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:49:22.2498064Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] EngineCore failed to start. 2025-10-10T01:49:22.2498639Z 2025-10-10T01:49:22.2499152Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] Traceback (most recent call last): 2025-10-10T01:49:22.2499671Z 2025-10-10T01:49:22.2500371Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:22.2501029Z 2025-10-10T01:49:22.2501425Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:22.2501805Z 2025-10-10T01:49:22.2502329Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2502627Z 2025-10-10T01:49:22.2503161Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:22.2503624Z 2025-10-10T01:49:22.2504036Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:22.2504420Z 2025-10-10T01:49:22.2505043Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:22.2505495Z 2025-10-10T01:49:22.2505825Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:22.2506133Z 2025-10-10T01:49:22.2506409Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2506686Z 2025-10-10T01:49:22.2507183Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:22.2507763Z 2025-10-10T01:49:22.2508128Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] self._init_executor() 2025-10-10T01:49:22.2508413Z 2025-10-10T01:49:22.2509403Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:22.2510188Z 2025-10-10T01:49:22.2510852Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:22.2511257Z 2025-10-10T01:49:22.2511992Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:22.2512497Z 2025-10-10T01:49:22.2512862Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:22.2513204Z 2025-10-10T01:49:22.2513506Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2513789Z 2025-10-10T01:49:22.2514289Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:22.2514738Z 2025-10-10T01:49:22.2515126Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:49:22.2515401Z 2025-10-10T01:49:22.2515650Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2515910Z 2025-10-10T01:49:22.2516422Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:22.2516920Z 2025-10-10T01:49:22.2517213Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:22.2517502Z 2025-10-10T01:49:22.2517775Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2518041Z 2025-10-10T01:49:22.2518592Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:22.2519190Z 2025-10-10T01:49:22.2519557Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:49:22.2519861Z 2025-10-10T01:49:22.2520139Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2520415Z 2025-10-10T01:49:22.2520894Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:22.2521311Z 2025-10-10T01:49:22.2521675Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:22.2522003Z 2025-10-10T01:49:22.2522297Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2522577Z 2025-10-10T01:49:22.2522926Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:49:22.2523253Z 2025-10-10T01:49:22.2523597Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:49:22.2523933Z 2025-10-10T01:49:22.2524304Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:22.2524723Z 2025-10-10T01:49:22.2525065Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:49:22.2525398Z 2025-10-10T01:49:22.2525796Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:49:22.2526146Z 2025-10-10T01:49:22.2526523Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:22.2526878Z 2025-10-10T01:49:22.2527376Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:22.2527851Z 2025-10-10T01:49:22.2528208Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:22.2528538Z 2025-10-10T01:49:22.2529061Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:22.2529590Z 2025-10-10T01:49:22.2529965Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:22.2530324Z 2025-10-10T01:49:22.2530875Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:22.2531392Z 2025-10-10T01:49:22.2531678Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:49:22.2531974Z 2025-10-10T01:49:22.2532601Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:22.2533175Z 2025-10-10T01:49:22.2533537Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:22.2533886Z 2025-10-10T01:49:22.2534190Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2534454Z 2025-10-10T01:49:22.2535016Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:22.2535527Z 2025-10-10T01:49:22.2535821Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:49:22.2536112Z 2025-10-10T01:49:22.2536389Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2536665Z 2025-10-10T01:49:22.2537261Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:22.2537811Z 2025-10-10T01:49:22.2538092Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:49:22.2538378Z 2025-10-10T01:49:22.2538632Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2538902Z 2025-10-10T01:49:22.2539474Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:22.2540044Z 2025-10-10T01:49:22.2540358Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:22.2540662Z 2025-10-10T01:49:22.2540973Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2541246Z 2025-10-10T01:49:22.2541769Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:22.2542248Z 2025-10-10T01:49:22.2542531Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:49:22.2542811Z 2025-10-10T01:49:22.2543070Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2543340Z 2025-10-10T01:49:22.2543853Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:22.2544334Z 2025-10-10T01:49:22.2544642Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:49:22.2544990Z 2025-10-10T01:49:22.2545213Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:49:22.2545451Z 2025-10-10T01:49:22.2545936Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:22.2546382Z 2025-10-10T01:49:22.2546632Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] raise RuntimeError( 2025-10-10T01:49:22.2546896Z 2025-10-10T01:49:22.2547474Z (EngineCore_DP0 pid=6693) ERROR 10-10 01:49:22 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:22.2548159Z (EngineCore_DP0 pid=6693) Process EngineCore_DP0: 2025-10-10T01:49:22.2548555Z (EngineCore_DP0 pid=6693) Traceback (most recent call last): 2025-10-10T01:49:22.2549203Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:49:22.2549717Z (EngineCore_DP0 pid=6693) self.run() 2025-10-10T01:49:22.2550270Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:49:22.2550835Z (EngineCore_DP0 pid=6693) self._target(*self._args, **self._kwargs) 2025-10-10T01:49:22.2551511Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:49:22.2552065Z (EngineCore_DP0 pid=6693) raise e 2025-10-10T01:49:22.2552661Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:22.2553296Z (EngineCore_DP0 pid=6693) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:22.2553737Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2554357Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:22.2555011Z (EngineCore_DP0 pid=6693) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:22.2555663Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:22.2556348Z (EngineCore_DP0 pid=6693) self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:22.2556818Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2557536Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:22.2558127Z (EngineCore_DP0 pid=6693) self._init_executor() 2025-10-10T01:49:22.2558782Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:22.2559528Z (EngineCore_DP0 pid=6693) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:22.2560287Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:22.2561008Z (EngineCore_DP0 pid=6693) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:22.2561498Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2562199Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:22.2562777Z (EngineCore_DP0 pid=6693) return func(*args, **kwargs) 2025-10-10T01:49:22.2563157Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2563794Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:22.2564430Z (EngineCore_DP0 pid=6693) worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:22.2564854Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2565517Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:22.2566170Z (EngineCore_DP0 pid=6693) module = importlib.import_module(module_name) 2025-10-10T01:49:22.2566658Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2567243Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:22.2567863Z (EngineCore_DP0 pid=6693) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:22.2568347Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2568838Z (EngineCore_DP0 pid=6693) File "", line 1387, in _gcd_import 2025-10-10T01:49:22.2569394Z (EngineCore_DP0 pid=6693) File "", line 1360, in _find_and_load 2025-10-10T01:49:22.2569978Z (EngineCore_DP0 pid=6693) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:22.2570560Z (EngineCore_DP0 pid=6693) File "", line 935, in _load_unlocked 2025-10-10T01:49:22.2571131Z (EngineCore_DP0 pid=6693) File "", line 999, in exec_module 2025-10-10T01:49:22.2571732Z (EngineCore_DP0 pid=6693) File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:22.2572459Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:22.2573134Z (EngineCore_DP0 pid=6693) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:22.2573894Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:22.2574646Z (EngineCore_DP0 pid=6693) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:22.2575435Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:22.2576104Z (EngineCore_DP0 pid=6693) class FlashAttentionMetadataBuilder( 2025-10-10T01:49:22.2576893Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:22.2577717Z (EngineCore_DP0 pid=6693) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:22.2578200Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2578907Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:22.2579645Z (EngineCore_DP0 pid=6693) if not is_fa_version_supported(fa_version): 2025-10-10T01:49:22.2580089Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2580844Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:22.2581566Z (EngineCore_DP0 pid=6693) return _is_fa2_supported(device)[0] 2025-10-10T01:49:22.2581973Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2582687Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:22.2583401Z (EngineCore_DP0 pid=6693) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:22.2583856Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2584582Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:22.2585224Z (EngineCore_DP0 pid=6693) prop = get_device_properties(device) 2025-10-10T01:49:22.2585642Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:22.2586299Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:22.2586955Z (EngineCore_DP0 pid=6693) _lazy_init() # will define _get_device_properties 2025-10-10T01:49:22.2587349Z (EngineCore_DP0 pid=6693) ^^^^^^^^^^^^ 2025-10-10T01:49:22.2587919Z (EngineCore_DP0 pid=6693) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:22.2588493Z (EngineCore_DP0 pid=6693) raise RuntimeError( 2025-10-10T01:49:22.2589178Z (EngineCore_DP0 pid=6693) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:22.6794860Z FAILED 2025-10-10T01:49:22.6926249Z models/test_initialization.py::test_can_initialize_large_subset[ExaoneForCausalLM] Fork a new process to run a test 6697 2025-10-10T01:49:22.6937653Z Fork a new process to run a test 0 2025-10-10T01:49:22.7211071Z INFO 10-10 01:49:22 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ExaoneForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct'} 2025-10-10T01:49:22.8212264Z 2025-10-10T01:49:22.8213646Z config.json: 0% 0.00/878 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:49:30.7960079Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:49:30.7960566Z 2025-10-10T01:49:30.7961068Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:49:30.7961377Z 2025-10-10T01:49:30.7961938Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:30.7962425Z 2025-10-10T01:49:30.7962733Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:30.7963057Z 2025-10-10T01:49:30.7963335Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7963622Z 2025-10-10T01:49:30.7964103Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:30.7964553Z 2025-10-10T01:49:30.7964995Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:30.7965329Z 2025-10-10T01:49:30.7965800Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:30.7966234Z 2025-10-10T01:49:30.7966593Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:30.7967044Z 2025-10-10T01:49:30.7967338Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7967772Z 2025-10-10T01:49:30.7968328Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:30.7968807Z 2025-10-10T01:49:30.7969070Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] self._init_executor() 2025-10-10T01:49:30.7969332Z 2025-10-10T01:49:30.7969882Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:30.7970604Z 2025-10-10T01:49:30.7970955Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:30.7971455Z 2025-10-10T01:49:30.7972122Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:30.7973189Z 2025-10-10T01:49:30.7973963Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:30.7974568Z 2025-10-10T01:49:30.7974905Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7975204Z 2025-10-10T01:49:30.7975704Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:30.7976172Z 2025-10-10T01:49:30.7976444Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:49:30.7976711Z 2025-10-10T01:49:30.7976966Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7977231Z 2025-10-10T01:49:30.7977769Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:30.7978311Z 2025-10-10T01:49:30.7978611Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:30.7978903Z 2025-10-10T01:49:30.7979177Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7979447Z 2025-10-10T01:49:30.7979972Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:30.7980470Z 2025-10-10T01:49:30.7980773Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:49:30.7981078Z 2025-10-10T01:49:30.7981347Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7981621Z 2025-10-10T01:49:30.7982214Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:30.7982707Z 2025-10-10T01:49:30.7983076Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:30.7983407Z 2025-10-10T01:49:30.7983698Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7983973Z 2025-10-10T01:49:30.7984318Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:49:30.7984647Z 2025-10-10T01:49:30.7984988Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:49:30.7985330Z 2025-10-10T01:49:30.7985701Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:30.7986061Z 2025-10-10T01:49:30.7986401Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:49:30.7986733Z 2025-10-10T01:49:30.7987087Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:49:30.7987432Z 2025-10-10T01:49:30.7987805Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:30.7988236Z 2025-10-10T01:49:30.7988784Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:30.7989248Z 2025-10-10T01:49:30.7989608Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:30.7989960Z 2025-10-10T01:49:30.7990509Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:30.7990990Z 2025-10-10T01:49:30.7991361Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:30.7991737Z 2025-10-10T01:49:30.7992289Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:30.7992808Z 2025-10-10T01:49:30.7993159Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:49:30.7993459Z 2025-10-10T01:49:30.7994101Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:30.7994682Z 2025-10-10T01:49:30.7995045Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:30.7995395Z 2025-10-10T01:49:30.7995653Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7995918Z 2025-10-10T01:49:30.7996727Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:30.7997253Z 2025-10-10T01:49:30.7997633Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:49:30.7997936Z 2025-10-10T01:49:30.7998211Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.7998487Z 2025-10-10T01:49:30.7999207Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:30.7999771Z 2025-10-10T01:49:30.8000057Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:49:30.8000346Z 2025-10-10T01:49:30.8000601Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8000877Z 2025-10-10T01:49:30.8001460Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:30.8001983Z 2025-10-10T01:49:30.8002307Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:30.8002616Z 2025-10-10T01:49:30.8002892Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8003163Z 2025-10-10T01:49:30.8003690Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:30.8004281Z 2025-10-10T01:49:30.8004569Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:49:30.8004912Z 2025-10-10T01:49:30.8005183Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8005469Z 2025-10-10T01:49:30.8005987Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:30.8006472Z 2025-10-10T01:49:30.8006779Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:49:30.8007099Z 2025-10-10T01:49:30.8007326Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:49:30.8007572Z 2025-10-10T01:49:30.8008068Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:30.8008609Z 2025-10-10T01:49:30.8008877Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] raise RuntimeError( 2025-10-10T01:49:30.8009133Z 2025-10-10T01:49:30.8009706Z (EngineCore_DP0 pid=6755) ERROR 10-10 01:49:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:30.8010396Z (EngineCore_DP0 pid=6755) Process EngineCore_DP0: 2025-10-10T01:49:30.8010811Z (EngineCore_DP0 pid=6755) Traceback (most recent call last): 2025-10-10T01:49:30.8011414Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:49:30.8011933Z (EngineCore_DP0 pid=6755) self.run() 2025-10-10T01:49:30.8012458Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:49:30.8019848Z (EngineCore_DP0 pid=6755) self._target(*self._args, **self._kwargs) 2025-10-10T01:49:30.8020761Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:49:30.8021349Z (EngineCore_DP0 pid=6755) raise e 2025-10-10T01:49:30.8021978Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:30.8022624Z (EngineCore_DP0 pid=6755) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:30.8023085Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8023717Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:30.8024375Z (EngineCore_DP0 pid=6755) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:30.8025054Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:30.8025670Z (EngineCore_DP0 pid=6755) self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:30.8026148Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8026821Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:30.8027487Z (EngineCore_DP0 pid=6755) self._init_executor() 2025-10-10T01:49:30.8028163Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:30.8028895Z (EngineCore_DP0 pid=6755) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:30.8029645Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:30.8030361Z (EngineCore_DP0 pid=6755) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:30.8030858Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8031499Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:30.8032108Z (EngineCore_DP0 pid=6755) return func(*args, **kwargs) 2025-10-10T01:49:30.8032490Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8033154Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:30.8033857Z (EngineCore_DP0 pid=6755) worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:30.8034291Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8034962Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:30.8035611Z (EngineCore_DP0 pid=6755) module = importlib.import_module(module_name) 2025-10-10T01:49:30.8036059Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8036644Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:30.8037273Z (EngineCore_DP0 pid=6755) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:30.8037775Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8038309Z (EngineCore_DP0 pid=6755) File "", line 1387, in _gcd_import 2025-10-10T01:49:30.8038870Z (EngineCore_DP0 pid=6755) File "", line 1360, in _find_and_load 2025-10-10T01:49:30.8039569Z (EngineCore_DP0 pid=6755) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:30.8040152Z (EngineCore_DP0 pid=6755) File "", line 935, in _load_unlocked 2025-10-10T01:49:30.8040727Z (EngineCore_DP0 pid=6755) File "", line 999, in exec_module 2025-10-10T01:49:30.8041323Z (EngineCore_DP0 pid=6755) File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:30.8042057Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:30.8042748Z (EngineCore_DP0 pid=6755) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:30.8043481Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:30.8044208Z (EngineCore_DP0 pid=6755) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:30.8045007Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:30.8045761Z (EngineCore_DP0 pid=6755) class FlashAttentionMetadataBuilder( 2025-10-10T01:49:30.8046598Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:30.8047426Z (EngineCore_DP0 pid=6755) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:30.8047908Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8048600Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:30.8049277Z (EngineCore_DP0 pid=6755) if not is_fa_version_supported(fa_version): 2025-10-10T01:49:30.8049722Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8050473Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:30.8051181Z (EngineCore_DP0 pid=6755) return _is_fa2_supported(device)[0] 2025-10-10T01:49:30.8051651Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8052370Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:30.8053092Z (EngineCore_DP0 pid=6755) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:30.8053546Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8054219Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:30.8054863Z (EngineCore_DP0 pid=6755) prop = get_device_properties(device) 2025-10-10T01:49:30.8055285Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:30.8055988Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:30.8056660Z (EngineCore_DP0 pid=6755) _lazy_init() # will define _get_device_properties 2025-10-10T01:49:30.8057055Z (EngineCore_DP0 pid=6755) ^^^^^^^^^^^^ 2025-10-10T01:49:30.8057635Z (EngineCore_DP0 pid=6755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:30.8058197Z (EngineCore_DP0 pid=6755) raise RuntimeError( 2025-10-10T01:49:30.8058873Z (EngineCore_DP0 pid=6755) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:31.2055347Z FAILED 2025-10-10T01:49:31.2185682Z models/test_initialization.py::test_can_initialize_large_subset[RWForCausalLM] Fork a new process to run a test 6759 2025-10-10T01:49:31.2197646Z Fork a new process to run a test 0 2025-10-10T01:49:31.2477969Z INFO 10-10 01:49:31 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='RWForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'tiiuae/falcon-40b'} 2025-10-10T01:49:31.3202974Z 2025-10-10T01:49:31.3205201Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:49:31.3205511Z config.json: 1.05kB [00:00, 5.00MB/s] 2025-10-10T01:49:38.3969039Z INFO 10-10 01:49:38 [model.py:551] Resolved architecture: FalconForCausalLM 2025-10-10T01:49:38.4216075Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:49:38.4216506Z INFO 10-10 01:49:38 [model.py:1545] Using max model len 2048 2025-10-10T01:49:38.5952693Z INFO 10-10 01:49:38 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:49:38.6319324Z 2025-10-10T01:49:38.6320494Z tokenizer_config.json: 0% 0.00/242 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:49:39.2344104Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] EngineCore failed to start. 2025-10-10T01:49:39.2344516Z 2025-10-10T01:49:39.2344903Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] Traceback (most recent call last): 2025-10-10T01:49:39.2345296Z 2025-10-10T01:49:39.2345975Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:39.2346594Z 2025-10-10T01:49:39.2347004Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:39.2347561Z 2025-10-10T01:49:39.2347857Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2348134Z 2025-10-10T01:49:39.2348724Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:39.2349186Z 2025-10-10T01:49:39.2349526Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:39.2349850Z 2025-10-10T01:49:39.2350323Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:39.2350773Z 2025-10-10T01:49:39.2351118Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:39.2351441Z 2025-10-10T01:49:39.2351718Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2352000Z 2025-10-10T01:49:39.2352608Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:39.2353083Z 2025-10-10T01:49:39.2353351Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] self._init_executor() 2025-10-10T01:49:39.2353666Z 2025-10-10T01:49:39.2354611Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:39.2355407Z 2025-10-10T01:49:39.2356069Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:39.2356660Z 2025-10-10T01:49:39.2357719Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:39.2358267Z 2025-10-10T01:49:39.2358771Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:39.2359254Z 2025-10-10T01:49:39.2359561Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2359849Z 2025-10-10T01:49:39.2360340Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:39.2360803Z 2025-10-10T01:49:39.2361066Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:49:39.2361353Z 2025-10-10T01:49:39.2361604Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2361865Z 2025-10-10T01:49:39.2362388Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:39.2362862Z 2025-10-10T01:49:39.2363154Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:39.2363441Z 2025-10-10T01:49:39.2363706Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2363972Z 2025-10-10T01:49:39.2364492Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:39.2365052Z 2025-10-10T01:49:39.2365357Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:49:39.2365723Z 2025-10-10T01:49:39.2366006Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2366286Z 2025-10-10T01:49:39.2366729Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:39.2367143Z 2025-10-10T01:49:39.2367478Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:39.2367808Z 2025-10-10T01:49:39.2368106Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2368387Z 2025-10-10T01:49:39.2368732Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:49:39.2369128Z 2025-10-10T01:49:39.2369483Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:49:39.2369815Z 2025-10-10T01:49:39.2370188Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:39.2370551Z 2025-10-10T01:49:39.2370887Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:49:39.2371224Z 2025-10-10T01:49:39.2371578Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:49:39.2371953Z 2025-10-10T01:49:39.2372331Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:39.2372692Z 2025-10-10T01:49:39.2373253Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:39.2373716Z 2025-10-10T01:49:39.2374073Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:39.2374403Z 2025-10-10T01:49:39.2374925Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:39.2375403Z 2025-10-10T01:49:39.2375782Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:39.2376145Z 2025-10-10T01:49:39.2376717Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:39.2377245Z 2025-10-10T01:49:39.2377536Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:49:39.2377836Z 2025-10-10T01:49:39.2378460Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:39.2379055Z 2025-10-10T01:49:39.2379421Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:39.2379829Z 2025-10-10T01:49:39.2380086Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2380350Z 2025-10-10T01:49:39.2380978Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:39.2381506Z 2025-10-10T01:49:39.2381823Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:49:39.2382119Z 2025-10-10T01:49:39.2382394Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2382663Z 2025-10-10T01:49:39.2383259Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:39.2383818Z 2025-10-10T01:49:39.2384095Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:49:39.2384432Z 2025-10-10T01:49:39.2384695Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2384971Z 2025-10-10T01:49:39.2385546Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:39.2386078Z 2025-10-10T01:49:39.2386393Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:39.2386715Z 2025-10-10T01:49:39.2386993Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2387267Z 2025-10-10T01:49:39.2387796Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:39.2388282Z 2025-10-10T01:49:39.2388614Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:49:39.2388899Z 2025-10-10T01:49:39.2389166Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2389433Z 2025-10-10T01:49:39.2389961Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:39.2390450Z 2025-10-10T01:49:39.2390771Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:49:39.2391080Z 2025-10-10T01:49:39.2391303Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:49:39.2391550Z 2025-10-10T01:49:39.2392035Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:39.2392488Z 2025-10-10T01:49:39.2392739Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] raise RuntimeError( 2025-10-10T01:49:39.2392993Z 2025-10-10T01:49:39.2393560Z (EngineCore_DP0 pid=6817) ERROR 10-10 01:49:39 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:39.2394254Z (EngineCore_DP0 pid=6817) Process EngineCore_DP0: 2025-10-10T01:49:39.2394726Z (EngineCore_DP0 pid=6817) Traceback (most recent call last): 2025-10-10T01:49:39.2395360Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:49:39.2395920Z (EngineCore_DP0 pid=6817) self.run() 2025-10-10T01:49:39.2396821Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:49:39.2397436Z (EngineCore_DP0 pid=6817) self._target(*self._args, **self._kwargs) 2025-10-10T01:49:39.2398112Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:49:39.2398655Z (EngineCore_DP0 pid=6817) raise e 2025-10-10T01:49:39.2399367Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:39.2400017Z (EngineCore_DP0 pid=6817) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:39.2400460Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2401227Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:39.2401872Z (EngineCore_DP0 pid=6817) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:39.2402531Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:39.2403147Z (EngineCore_DP0 pid=6817) self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:39.2403601Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2404261Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:39.2404849Z (EngineCore_DP0 pid=6817) self._init_executor() 2025-10-10T01:49:39.2405512Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:39.2406269Z (EngineCore_DP0 pid=6817) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:39.2407007Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:39.2407730Z (EngineCore_DP0 pid=6817) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:39.2408228Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2408868Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:39.2409472Z (EngineCore_DP0 pid=6817) return func(*args, **kwargs) 2025-10-10T01:49:39.2409864Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2410508Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:39.2411132Z (EngineCore_DP0 pid=6817) worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:39.2411552Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2412218Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:39.2412948Z (EngineCore_DP0 pid=6817) module = importlib.import_module(module_name) 2025-10-10T01:49:39.2413390Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2414026Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:39.2414655Z (EngineCore_DP0 pid=6817) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:39.2415152Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2415649Z (EngineCore_DP0 pid=6817) File "", line 1387, in _gcd_import 2025-10-10T01:49:39.2416203Z (EngineCore_DP0 pid=6817) File "", line 1360, in _find_and_load 2025-10-10T01:49:39.2416820Z (EngineCore_DP0 pid=6817) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:39.2417395Z (EngineCore_DP0 pid=6817) File "", line 935, in _load_unlocked 2025-10-10T01:49:39.2417966Z (EngineCore_DP0 pid=6817) File "", line 999, in exec_module 2025-10-10T01:49:39.2418623Z (EngineCore_DP0 pid=6817) File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:39.2419359Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:39.2420043Z (EngineCore_DP0 pid=6817) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:39.2420776Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:39.2421493Z (EngineCore_DP0 pid=6817) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:39.2422274Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:39.2422940Z (EngineCore_DP0 pid=6817) class FlashAttentionMetadataBuilder( 2025-10-10T01:49:39.2423812Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:39.2424652Z (EngineCore_DP0 pid=6817) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:39.2425133Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2425832Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:39.2426524Z (EngineCore_DP0 pid=6817) if not is_fa_version_supported(fa_version): 2025-10-10T01:49:39.2426965Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2427708Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:39.2428416Z (EngineCore_DP0 pid=6817) return _is_fa2_supported(device)[0] 2025-10-10T01:49:39.2428821Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2429533Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:39.2430247Z (EngineCore_DP0 pid=6817) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:39.2430748Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2431418Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:39.2432091Z (EngineCore_DP0 pid=6817) prop = get_device_properties(device) 2025-10-10T01:49:39.2432535Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:39.2433193Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:39.2433852Z (EngineCore_DP0 pid=6817) _lazy_init() # will define _get_device_properties 2025-10-10T01:49:39.2434246Z (EngineCore_DP0 pid=6817) ^^^^^^^^^^^^ 2025-10-10T01:49:39.2434821Z (EngineCore_DP0 pid=6817) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:39.2435383Z (EngineCore_DP0 pid=6817) raise RuntimeError( 2025-10-10T01:49:39.2436067Z (EngineCore_DP0 pid=6817) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:39.6408568Z FAILED 2025-10-10T01:49:39.6540040Z models/test_initialization.py::test_can_initialize_large_subset[ApertusForCausalLM] Fork a new process to run a test 6821 2025-10-10T01:49:39.6550473Z Fork a new process to run a test 0 2025-10-10T01:49:39.6830153Z INFO 10-10 01:49:39 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ApertusForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'swiss-ai/Apertus-8B-2509'} 2025-10-10T01:49:39.7955894Z 2025-10-10T01:49:39.7959008Z config.json: 0% 0.00/899 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:49:48.3357569Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] EngineCore failed to start. 2025-10-10T01:49:48.3357992Z 2025-10-10T01:49:48.3358382Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] Traceback (most recent call last): 2025-10-10T01:49:48.3358770Z 2025-10-10T01:49:48.3359619Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:48.3360228Z 2025-10-10T01:49:48.3360814Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:48.3361210Z 2025-10-10T01:49:48.3361565Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3361927Z 2025-10-10T01:49:48.3362754Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:48.3363335Z 2025-10-10T01:49:48.3363785Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:48.3364195Z 2025-10-10T01:49:48.3364802Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:48.3365390Z 2025-10-10T01:49:48.3366002Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:48.3366580Z 2025-10-10T01:49:48.3367136Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3367636Z 2025-10-10T01:49:48.3368565Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:48.3369311Z 2025-10-10T01:49:48.3369605Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] self._init_executor() 2025-10-10T01:49:48.3369873Z 2025-10-10T01:49:48.3370522Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:48.3371037Z 2025-10-10T01:49:48.3371360Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:48.3371687Z 2025-10-10T01:49:48.3372230Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:48.3372729Z 2025-10-10T01:49:48.3373067Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:48.3373407Z 2025-10-10T01:49:48.3373691Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3373974Z 2025-10-10T01:49:48.3374457Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:48.3374994Z 2025-10-10T01:49:48.3375261Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:49:48.3375533Z 2025-10-10T01:49:48.3375793Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3376058Z 2025-10-10T01:49:48.3376568Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:48.3377052Z 2025-10-10T01:49:48.3377339Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:48.3377630Z 2025-10-10T01:49:48.3377894Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3378170Z 2025-10-10T01:49:48.3378735Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:48.3379221Z 2025-10-10T01:49:48.3379522Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:49:48.3379823Z 2025-10-10T01:49:48.3380102Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3380379Z 2025-10-10T01:49:48.3380831Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:48.3381244Z 2025-10-10T01:49:48.3381630Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:48.3381966Z 2025-10-10T01:49:48.3382251Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3382525Z 2025-10-10T01:49:48.3382871Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:49:48.3383197Z 2025-10-10T01:49:48.3383543Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:49:48.3383928Z 2025-10-10T01:49:48.3384306Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:48.3384658Z 2025-10-10T01:49:48.3385043Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:49:48.3385380Z 2025-10-10T01:49:48.3385741Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:49:48.3386104Z 2025-10-10T01:49:48.3386473Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:48.3386834Z 2025-10-10T01:49:48.3387325Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:48.3387787Z 2025-10-10T01:49:48.3388135Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:48.3388468Z 2025-10-10T01:49:48.3389095Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:48.3389577Z 2025-10-10T01:49:48.3389949Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:48.3390305Z 2025-10-10T01:49:48.3390859Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:48.3391359Z 2025-10-10T01:49:48.3391656Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:49:48.3391945Z 2025-10-10T01:49:48.3392566Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:48.3393139Z 2025-10-10T01:49:48.3393548Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:48.3393912Z 2025-10-10T01:49:48.3394178Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3394446Z 2025-10-10T01:49:48.3395011Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:48.3395534Z 2025-10-10T01:49:48.3395825Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:49:48.3396315Z 2025-10-10T01:49:48.3396612Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3396886Z 2025-10-10T01:49:48.3397495Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:48.3398040Z 2025-10-10T01:49:48.3398324Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:49:48.3398602Z 2025-10-10T01:49:48.3398857Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3399327Z 2025-10-10T01:49:48.3399913Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:48.3400447Z 2025-10-10T01:49:48.3400829Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:48.3401165Z 2025-10-10T01:49:48.3401443Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3401718Z 2025-10-10T01:49:48.3402236Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:48.3402713Z 2025-10-10T01:49:48.3402997Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:49:48.3403281Z 2025-10-10T01:49:48.3403542Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3403806Z 2025-10-10T01:49:48.3404329Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:48.3404871Z 2025-10-10T01:49:48.3405184Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:49:48.3405487Z 2025-10-10T01:49:48.3405719Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:49:48.3405967Z 2025-10-10T01:49:48.3406447Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:48.3406902Z 2025-10-10T01:49:48.3407144Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] raise RuntimeError( 2025-10-10T01:49:48.3407398Z 2025-10-10T01:49:48.3407975Z (EngineCore_DP0 pid=6899) ERROR 10-10 01:49:48 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:48.3408735Z (EngineCore_DP0 pid=6899) Process EngineCore_DP0: 2025-10-10T01:49:48.3409145Z (EngineCore_DP0 pid=6899) Traceback (most recent call last): 2025-10-10T01:49:48.3409745Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:49:48.3410262Z (EngineCore_DP0 pid=6899) self.run() 2025-10-10T01:49:48.3410782Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:49:48.3411344Z (EngineCore_DP0 pid=6899) self._target(*self._args, **self._kwargs) 2025-10-10T01:49:48.3412012Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:49:48.3412573Z (EngineCore_DP0 pid=6899) raise e 2025-10-10T01:49:48.3413199Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:48.3413841Z (EngineCore_DP0 pid=6899) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:48.3414284Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3414905Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:48.3415553Z (EngineCore_DP0 pid=6899) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:48.3416259Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:48.3416911Z (EngineCore_DP0 pid=6899) self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:48.3417370Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3418019Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:48.3418597Z (EngineCore_DP0 pid=6899) self._init_executor() 2025-10-10T01:49:48.3419255Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:48.3419939Z (EngineCore_DP0 pid=6899) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:48.3420676Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:48.3421389Z (EngineCore_DP0 pid=6899) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:48.3421926Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3422563Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:48.3423147Z (EngineCore_DP0 pid=6899) return func(*args, **kwargs) 2025-10-10T01:49:48.3423525Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3424172Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:48.3424807Z (EngineCore_DP0 pid=6899) worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:48.3425224Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3425947Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:48.3426610Z (EngineCore_DP0 pid=6899) module = importlib.import_module(module_name) 2025-10-10T01:49:48.3427061Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3427644Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:48.3428271Z (EngineCore_DP0 pid=6899) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:48.3428760Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3429259Z (EngineCore_DP0 pid=6899) File "", line 1387, in _gcd_import 2025-10-10T01:49:48.3429816Z (EngineCore_DP0 pid=6899) File "", line 1360, in _find_and_load 2025-10-10T01:49:48.3430420Z (EngineCore_DP0 pid=6899) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:48.3431000Z (EngineCore_DP0 pid=6899) File "", line 935, in _load_unlocked 2025-10-10T01:49:48.3431563Z (EngineCore_DP0 pid=6899) File "", line 999, in exec_module 2025-10-10T01:49:48.3432164Z (EngineCore_DP0 pid=6899) File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:48.3432894Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:48.3433630Z (EngineCore_DP0 pid=6899) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:48.3434383Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:48.3435114Z (EngineCore_DP0 pid=6899) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:48.3435902Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:48.3436569Z (EngineCore_DP0 pid=6899) class FlashAttentionMetadataBuilder( 2025-10-10T01:49:48.3437358Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:48.3438169Z (EngineCore_DP0 pid=6899) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:48.3438644Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3439436Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:48.3440118Z (EngineCore_DP0 pid=6899) if not is_fa_version_supported(fa_version): 2025-10-10T01:49:48.3440558Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3441298Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:48.3441993Z (EngineCore_DP0 pid=6899) return _is_fa2_supported(device)[0] 2025-10-10T01:49:48.3442398Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3443111Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:48.3443894Z (EngineCore_DP0 pid=6899) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:48.3444367Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3445041Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:48.3445675Z (EngineCore_DP0 pid=6899) prop = get_device_properties(device) 2025-10-10T01:49:48.3446082Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:48.3446746Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:48.3447404Z (EngineCore_DP0 pid=6899) _lazy_init() # will define _get_device_properties 2025-10-10T01:49:48.3447804Z (EngineCore_DP0 pid=6899) ^^^^^^^^^^^^ 2025-10-10T01:49:48.3448381Z (EngineCore_DP0 pid=6899) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:48.3448935Z (EngineCore_DP0 pid=6899) raise RuntimeError( 2025-10-10T01:49:48.3449617Z (EngineCore_DP0 pid=6899) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:48.7592959Z FAILED 2025-10-10T01:49:48.7723512Z models/test_initialization.py::test_can_initialize_large_subset[OlmoForCausalLM] Fork a new process to run a test 6903 2025-10-10T01:49:48.7734162Z Fork a new process to run a test 0 2025-10-10T01:49:48.8006174Z INFO 10-10 01:49:48 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='OlmoForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'allenai/OLMo-1B-hf'} 2025-10-10T01:49:48.9445812Z 2025-10-10T01:49:48.9447014Z config.json: 0% 0.00/632 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:49:56.5128913Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] EngineCore failed to start. 2025-10-10T01:49:56.5129345Z 2025-10-10T01:49:56.5129996Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] Traceback (most recent call last): 2025-10-10T01:49:56.5130397Z 2025-10-10T01:49:56.5131092Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:56.5131696Z 2025-10-10T01:49:56.5132085Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:56.5132482Z 2025-10-10T01:49:56.5132812Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5133102Z 2025-10-10T01:49:56.5133594Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:56.5134251Z 2025-10-10T01:49:56.5134921Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:56.5135288Z 2025-10-10T01:49:56.5135798Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:56.5136238Z 2025-10-10T01:49:56.5136556Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:56.5136861Z 2025-10-10T01:49:56.5137140Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5137416Z 2025-10-10T01:49:56.5138359Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:56.5139178Z 2025-10-10T01:49:56.5139928Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] self._init_executor() 2025-10-10T01:49:56.5140432Z 2025-10-10T01:49:56.5141268Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:56.5141828Z 2025-10-10T01:49:56.5142180Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:56.5142508Z 2025-10-10T01:49:56.5143058Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:56.5143588Z 2025-10-10T01:49:56.5143940Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:56.5144276Z 2025-10-10T01:49:56.5144572Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5144846Z 2025-10-10T01:49:56.5145336Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:56.5145784Z 2025-10-10T01:49:56.5146052Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:49:56.5146318Z 2025-10-10T01:49:56.5146698Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5146964Z 2025-10-10T01:49:56.5147526Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:56.5148008Z 2025-10-10T01:49:56.5148298Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:56.5148593Z 2025-10-10T01:49:56.5148855Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5149130Z 2025-10-10T01:49:56.5149651Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:56.5150130Z 2025-10-10T01:49:56.5150454Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:49:56.5150754Z 2025-10-10T01:49:56.5151029Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5151359Z 2025-10-10T01:49:56.5151805Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:56.5152211Z 2025-10-10T01:49:56.5152593Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:56.5152921Z 2025-10-10T01:49:56.5153210Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5153483Z 2025-10-10T01:49:56.5153824Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:49:56.5154151Z 2025-10-10T01:49:56.5154497Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:49:56.5154838Z 2025-10-10T01:49:56.5155251Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:56.5155628Z 2025-10-10T01:49:56.5155968Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:49:56.5156302Z 2025-10-10T01:49:56.5156658Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:49:56.5157009Z 2025-10-10T01:49:56.5157376Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:56.5157747Z 2025-10-10T01:49:56.5158240Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:56.5158698Z 2025-10-10T01:49:56.5159068Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:56.5159571Z 2025-10-10T01:49:56.5160111Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:56.5160587Z 2025-10-10T01:49:56.5160962Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:56.5161374Z 2025-10-10T01:49:56.5161935Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:56.5162442Z 2025-10-10T01:49:56.5162789Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:49:56.5163091Z 2025-10-10T01:49:56.5163725Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:56.5164302Z 2025-10-10T01:49:56.5164664Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:56.5165024Z 2025-10-10T01:49:56.5165282Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5165555Z 2025-10-10T01:49:56.5166127Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:56.5166734Z 2025-10-10T01:49:56.5167046Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:49:56.5167343Z 2025-10-10T01:49:56.5167622Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5167893Z 2025-10-10T01:49:56.5168498Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:56.5169048Z 2025-10-10T01:49:56.5169333Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:49:56.5169622Z 2025-10-10T01:49:56.5169880Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5170163Z 2025-10-10T01:49:56.5170792Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:56.5171329Z 2025-10-10T01:49:56.5171653Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:56.5171967Z 2025-10-10T01:49:56.5172264Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5172585Z 2025-10-10T01:49:56.5173217Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:56.5173791Z 2025-10-10T01:49:56.5174123Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:49:56.5174412Z 2025-10-10T01:49:56.5174681Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5174951Z 2025-10-10T01:49:56.5175473Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:56.5175964Z 2025-10-10T01:49:56.5176270Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:49:56.5176582Z 2025-10-10T01:49:56.5176806Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:49:56.5177103Z 2025-10-10T01:49:56.5177589Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:56.5178064Z 2025-10-10T01:49:56.5178373Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] raise RuntimeError( 2025-10-10T01:49:56.5178637Z 2025-10-10T01:49:56.5179206Z (EngineCore_DP0 pid=6961) ERROR 10-10 01:49:56 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:56.5179898Z (EngineCore_DP0 pid=6961) Process EngineCore_DP0: 2025-10-10T01:49:56.5180286Z (EngineCore_DP0 pid=6961) Traceback (most recent call last): 2025-10-10T01:49:56.5180887Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:49:56.5181407Z (EngineCore_DP0 pid=6961) self.run() 2025-10-10T01:49:56.5181929Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:49:56.5182546Z (EngineCore_DP0 pid=6961) self._target(*self._args, **self._kwargs) 2025-10-10T01:49:56.5183201Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:49:56.5183746Z (EngineCore_DP0 pid=6961) raise e 2025-10-10T01:49:56.5184331Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:49:56.5184963Z (EngineCore_DP0 pid=6961) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:49:56.5185404Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5186014Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:49:56.5186653Z (EngineCore_DP0 pid=6961) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:49:56.5187383Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:49:56.5188003Z (EngineCore_DP0 pid=6961) self.model_executor = executor_class(vllm_config) 2025-10-10T01:49:56.5188466Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5189113Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:49:56.5189705Z (EngineCore_DP0 pid=6961) self._init_executor() 2025-10-10T01:49:56.5190369Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:49:56.5191057Z (EngineCore_DP0 pid=6961) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:49:56.5191788Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:49:56.5192496Z (EngineCore_DP0 pid=6961) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:49:56.5192985Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5193633Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:49:56.5194290Z (EngineCore_DP0 pid=6961) return func(*args, **kwargs) 2025-10-10T01:49:56.5194678Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5195362Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:49:56.5195999Z (EngineCore_DP0 pid=6961) worker_class = resolve_obj_by_qualname( 2025-10-10T01:49:56.5196810Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5197496Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:49:56.5198153Z (EngineCore_DP0 pid=6961) module = importlib.import_module(module_name) 2025-10-10T01:49:56.5198599Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5199284Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:49:56.5199920Z (EngineCore_DP0 pid=6961) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:49:56.5200544Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5201254Z (EngineCore_DP0 pid=6961) File "", line 1387, in _gcd_import 2025-10-10T01:49:56.5201822Z (EngineCore_DP0 pid=6961) File "", line 1360, in _find_and_load 2025-10-10T01:49:56.5202404Z (EngineCore_DP0 pid=6961) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:49:56.5202989Z (EngineCore_DP0 pid=6961) File "", line 935, in _load_unlocked 2025-10-10T01:49:56.5203560Z (EngineCore_DP0 pid=6961) File "", line 999, in exec_module 2025-10-10T01:49:56.5204160Z (EngineCore_DP0 pid=6961) File "", line 488, in _call_with_frames_removed 2025-10-10T01:49:56.5204902Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:49:56.5205709Z (EngineCore_DP0 pid=6961) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:49:56.5206442Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:49:56.5207188Z (EngineCore_DP0 pid=6961) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:49:56.5207971Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:49:56.5208647Z (EngineCore_DP0 pid=6961) class FlashAttentionMetadataBuilder( 2025-10-10T01:49:56.5209440Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:49:56.5210260Z (EngineCore_DP0 pid=6961) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:49:56.5210736Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5211428Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:49:56.5212128Z (EngineCore_DP0 pid=6961) if not is_fa_version_supported(fa_version): 2025-10-10T01:49:56.5212574Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5213416Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:49:56.5214176Z (EngineCore_DP0 pid=6961) return _is_fa2_supported(device)[0] 2025-10-10T01:49:56.5214598Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5215312Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:49:56.5216041Z (EngineCore_DP0 pid=6961) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:49:56.5216494Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5217154Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:49:56.5217792Z (EngineCore_DP0 pid=6961) prop = get_device_properties(device) 2025-10-10T01:49:56.5218207Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:49:56.5218918Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:49:56.5219579Z (EngineCore_DP0 pid=6961) _lazy_init() # will define _get_device_properties 2025-10-10T01:49:56.5219967Z (EngineCore_DP0 pid=6961) ^^^^^^^^^^^^ 2025-10-10T01:49:56.5220548Z (EngineCore_DP0 pid=6961) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:49:56.5221105Z (EngineCore_DP0 pid=6961) raise RuntimeError( 2025-10-10T01:49:56.5221787Z (EngineCore_DP0 pid=6961) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:49:56.9228265Z FAILED 2025-10-10T01:49:56.9358686Z models/test_initialization.py::test_can_initialize_large_subset[PixtralForConditionalGeneration] Fork a new process to run a test 6965 2025-10-10T01:49:56.9371124Z Fork a new process to run a test 0 2025-10-10T01:49:56.9641248Z INFO 10-10 01:49:56 [utils.py:233] non-default args: {'tokenizer_mode': 'mistral', 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='PixtralForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mistralai/Pixtral-12B-2409'} 2025-10-10T01:49:57.0771487Z 2025-10-10T01:49:57.0773223Z params.json: 0% 0.00/552 [00:00", line 1387, in _gcd_import 2025-10-10T01:50:08.0114376Z 2025-10-10T01:50:08.0114586Z ERROR 10-10 01:50:08 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:50:08.0114865Z 2025-10-10T01:50:08.0115191Z ERROR 10-10 01:50:08 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:08.0115505Z 2025-10-10T01:50:08.0115706Z ERROR 10-10 01:50:08 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:50:08.0115985Z 2025-10-10T01:50:08.0116199Z ERROR 10-10 01:50:08 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:50:08.0116495Z 2025-10-10T01:50:08.0116721Z ERROR 10-10 01:50:08 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:08.0117032Z 2025-10-10T01:50:08.0117388Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:50:08.0117823Z 2025-10-10T01:50:08.0117995Z ERROR 10-10 01:50:08 [registry.py:542] from xformers import ops as xops 2025-10-10T01:50:08.0118216Z 2025-10-10T01:50:08.0118718Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:50:08.0119413Z 2025-10-10T01:50:08.0119579Z ERROR 10-10 01:50:08 [registry.py:542] from .fmha import ( 2025-10-10T01:50:08.0119862Z 2025-10-10T01:50:08.0120411Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:50:08.0120879Z 2025-10-10T01:50:08.0121111Z ERROR 10-10 01:50:08 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:50:08.0121539Z 2025-10-10T01:50:08.0121895Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:50:08.0122346Z 2025-10-10T01:50:08.0122546Z ERROR 10-10 01:50:08 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:50:08.0122787Z 2025-10-10T01:50:08.0122926Z ERROR 10-10 01:50:08 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:08.0123147Z 2025-10-10T01:50:08.0123472Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:50:08.0123853Z 2025-10-10T01:50:08.0123966Z ERROR 10-10 01:50:08 [registry.py:542] value = func() 2025-10-10T01:50:08.0124141Z 2025-10-10T01:50:08.0124246Z ERROR 10-10 01:50:08 [registry.py:542] ^^^^^^ 2025-10-10T01:50:08.0124415Z 2025-10-10T01:50:08.0124753Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:50:08.0125167Z 2025-10-10T01:50:08.0125346Z ERROR 10-10 01:50:08 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:50:08.0125645Z 2025-10-10T01:50:08.0125794Z ERROR 10-10 01:50:08 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:08.0126020Z 2025-10-10T01:50:08.0126372Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:08.0126793Z 2025-10-10T01:50:08.0126944Z ERROR 10-10 01:50:08 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:50:08.0127163Z 2025-10-10T01:50:08.0127295Z ERROR 10-10 01:50:08 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:08.0127497Z 2025-10-10T01:50:08.0127843Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:08.0128264Z 2025-10-10T01:50:08.0128435Z ERROR 10-10 01:50:08 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:08.0128687Z 2025-10-10T01:50:08.0128832Z ERROR 10-10 01:50:08 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:50:08.0129011Z 2025-10-10T01:50:08.0129329Z ERROR 10-10 01:50:08 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:08.0129717Z 2025-10-10T01:50:08.0129848Z ERROR 10-10 01:50:08 [registry.py:542] raise RuntimeError( 2025-10-10T01:50:08.0130040Z 2025-10-10T01:50:08.0130440Z ERROR 10-10 01:50:08 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:08.4038219Z FAILED 2025-10-10T01:50:08.4168093Z models/test_initialization.py::test_can_initialize_large_subset[LLaMAForCausalLM] Fork a new process to run a test 7043 2025-10-10T01:50:08.4180000Z Fork a new process to run a test 0 2025-10-10T01:50:08.4181715Z Model is not available online 2025-10-10T01:50:08.7215872Z PASSED 2025-10-10T01:50:08.7345725Z models/test_initialization.py::test_can_initialize_large_subset[JambaForCausalLM] Fork a new process to run a test 7044 2025-10-10T01:50:08.7356251Z Fork a new process to run a test 0 2025-10-10T01:50:08.7635500Z INFO 10-10 01:50:08 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='JambaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ai21labs/AI21-Jamba-1.5-Mini'} 2025-10-10T01:50:08.9836587Z 2025-10-10T01:50:08.9837477Z config.json: 0% 0.00/932 [00:00= mamba page size. 2025-10-10T01:50:16.0538071Z INFO 10-10 01:50:16 [config.py:397] Padding mamba page size by 5.26% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:50:16.1583717Z 2025-10-10T01:50:16.1585935Z tokenizer_config.json: 0% 0.00/14.2k [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. 2025-10-10T01:50:17.2896859Z 2025-10-10T01:50:17.2897266Z generation_config.json: 0% 0.00/126 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:50:17.4127492Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] EngineCore failed to start. 2025-10-10T01:50:17.4128284Z 2025-10-10T01:50:17.4128701Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] Traceback (most recent call last): 2025-10-10T01:50:17.4129079Z 2025-10-10T01:50:17.4129759Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:17.4130366Z 2025-10-10T01:50:17.4130760Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:17.4131137Z 2025-10-10T01:50:17.4131492Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4131834Z 2025-10-10T01:50:17.4132322Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:17.4132765Z 2025-10-10T01:50:17.4133301Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:17.4133785Z 2025-10-10T01:50:17.4134294Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:17.4134741Z 2025-10-10T01:50:17.4135053Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:17.4135369Z 2025-10-10T01:50:17.4135646Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4135922Z 2025-10-10T01:50:17.4136427Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:17.4136891Z 2025-10-10T01:50:17.4137216Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] self._init_executor() 2025-10-10T01:50:17.4137692Z 2025-10-10T01:50:17.4138746Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:17.4139534Z 2025-10-10T01:50:17.4140170Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:17.4140786Z 2025-10-10T01:50:17.4141379Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:17.4141897Z 2025-10-10T01:50:17.4142350Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:17.4142715Z 2025-10-10T01:50:17.4143014Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4143295Z 2025-10-10T01:50:17.4143802Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:17.4144266Z 2025-10-10T01:50:17.4144529Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:50:17.4144803Z 2025-10-10T01:50:17.4145056Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4145311Z 2025-10-10T01:50:17.4145830Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:17.4146365Z 2025-10-10T01:50:17.4146662Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:17.4146944Z 2025-10-10T01:50:17.4147208Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4147480Z 2025-10-10T01:50:17.4148045Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:17.4148529Z 2025-10-10T01:50:17.4148834Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:50:17.4149131Z 2025-10-10T01:50:17.4149416Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4149690Z 2025-10-10T01:50:17.4150175Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:17.4150584Z 2025-10-10T01:50:17.4150922Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:17.4151256Z 2025-10-10T01:50:17.4151537Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4151819Z 2025-10-10T01:50:17.4152167Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:50:17.4152513Z 2025-10-10T01:50:17.4152861Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:50:17.4153196Z 2025-10-10T01:50:17.4153592Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:17.4153950Z 2025-10-10T01:50:17.4154306Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:50:17.4154638Z 2025-10-10T01:50:17.4155001Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:50:17.4155394Z 2025-10-10T01:50:17.4155772Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:17.4156130Z 2025-10-10T01:50:17.4156667Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:17.4157142Z 2025-10-10T01:50:17.4157507Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:17.4164531Z 2025-10-10T01:50:17.4165179Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:17.4165715Z 2025-10-10T01:50:17.4166109Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:17.4166503Z 2025-10-10T01:50:17.4167082Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:17.4167711Z 2025-10-10T01:50:17.4168037Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:50:17.4168341Z 2025-10-10T01:50:17.4169012Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:17.4169595Z 2025-10-10T01:50:17.4169964Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:17.4170326Z 2025-10-10T01:50:17.4170585Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4170853Z 2025-10-10T01:50:17.4171455Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:17.4172001Z 2025-10-10T01:50:17.4172351Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:50:17.4172656Z 2025-10-10T01:50:17.4172930Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4173200Z 2025-10-10T01:50:17.4173808Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:17.4174355Z 2025-10-10T01:50:17.4174644Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:50:17.4174926Z 2025-10-10T01:50:17.4175197Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4175465Z 2025-10-10T01:50:17.4176047Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:17.4176585Z 2025-10-10T01:50:17.4176903Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:17.4177220Z 2025-10-10T01:50:17.4177492Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4177816Z 2025-10-10T01:50:17.4178339Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:17.4178824Z 2025-10-10T01:50:17.4179139Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:50:17.4179424Z 2025-10-10T01:50:17.4179695Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4179961Z 2025-10-10T01:50:17.4180481Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:17.4180959Z 2025-10-10T01:50:17.4181272Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:17.4181579Z 2025-10-10T01:50:17.4181804Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:50:17.4182048Z 2025-10-10T01:50:17.4182528Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:17.4183019Z 2025-10-10T01:50:17.4183267Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] raise RuntimeError( 2025-10-10T01:50:17.4183528Z 2025-10-10T01:50:17.4184092Z (EngineCore_DP0 pid=7123) ERROR 10-10 01:50:17 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:17.4184785Z (EngineCore_DP0 pid=7123) Process EngineCore_DP0: 2025-10-10T01:50:17.4185198Z (EngineCore_DP0 pid=7123) Traceback (most recent call last): 2025-10-10T01:50:17.4185806Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:50:17.4186318Z (EngineCore_DP0 pid=7123) self.run() 2025-10-10T01:50:17.4186845Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:50:17.4187452Z (EngineCore_DP0 pid=7123) self._target(*self._args, **self._kwargs) 2025-10-10T01:50:17.4188122Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:50:17.4188666Z (EngineCore_DP0 pid=7123) raise e 2025-10-10T01:50:17.4189250Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:17.4189883Z (EngineCore_DP0 pid=7123) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:17.4190326Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4190950Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:17.4191596Z (EngineCore_DP0 pid=7123) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:17.4192399Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:17.4193022Z (EngineCore_DP0 pid=7123) self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:17.4193475Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4194127Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:17.4194767Z (EngineCore_DP0 pid=7123) self._init_executor() 2025-10-10T01:50:17.4195419Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:17.4196371Z (EngineCore_DP0 pid=7123) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:17.4197152Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:17.4197886Z (EngineCore_DP0 pid=7123) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:17.4198395Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4199090Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:17.4199695Z (EngineCore_DP0 pid=7123) return func(*args, **kwargs) 2025-10-10T01:50:17.4200082Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4200816Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:17.4201448Z (EngineCore_DP0 pid=7123) worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:17.4201869Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4202880Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:17.4203569Z (EngineCore_DP0 pid=7123) module = importlib.import_module(module_name) 2025-10-10T01:50:17.4204028Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4204616Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:17.4205253Z (EngineCore_DP0 pid=7123) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:17.4205849Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4206366Z (EngineCore_DP0 pid=7123) File "", line 1387, in _gcd_import 2025-10-10T01:50:17.4206926Z (EngineCore_DP0 pid=7123) File "", line 1360, in _find_and_load 2025-10-10T01:50:17.4207520Z (EngineCore_DP0 pid=7123) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:17.4208099Z (EngineCore_DP0 pid=7123) File "", line 935, in _load_unlocked 2025-10-10T01:50:17.4208665Z (EngineCore_DP0 pid=7123) File "", line 999, in exec_module 2025-10-10T01:50:17.4209264Z (EngineCore_DP0 pid=7123) File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:17.4210001Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:17.4210681Z (EngineCore_DP0 pid=7123) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:17.4211401Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:17.4212115Z (EngineCore_DP0 pid=7123) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:17.4212985Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:17.4213654Z (EngineCore_DP0 pid=7123) class FlashAttentionMetadataBuilder( 2025-10-10T01:50:17.4214485Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:17.4215444Z (EngineCore_DP0 pid=7123) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:17.4215939Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4216631Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:17.4217339Z (EngineCore_DP0 pid=7123) if not is_fa_version_supported(fa_version): 2025-10-10T01:50:17.4217779Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4218698Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:17.4219524Z (EngineCore_DP0 pid=7123) return _is_fa2_supported(device)[0] 2025-10-10T01:50:17.4219942Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4220663Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:17.4221379Z (EngineCore_DP0 pid=7123) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:17.4221830Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4222546Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:17.4223192Z (EngineCore_DP0 pid=7123) prop = get_device_properties(device) 2025-10-10T01:50:17.4223620Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:17.4224339Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:17.4225018Z (EngineCore_DP0 pid=7123) _lazy_init() # will define _get_device_properties 2025-10-10T01:50:17.4225428Z (EngineCore_DP0 pid=7123) ^^^^^^^^^^^^ 2025-10-10T01:50:17.4226014Z (EngineCore_DP0 pid=7123) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:17.4226586Z (EngineCore_DP0 pid=7123) raise RuntimeError( 2025-10-10T01:50:17.4227277Z (EngineCore_DP0 pid=7123) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:17.8232117Z FAILED 2025-10-10T01:50:17.8362899Z models/test_initialization.py::test_can_initialize_large_subset[BailingMoeV2ForCausalLM] Fork a new process to run a test 7127 2025-10-10T01:50:17.8373368Z Fork a new process to run a test 0 2025-10-10T01:50:17.8653176Z INFO 10-10 01:50:17 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BailingMoeV2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'inclusionAI/Ling-mini-2.0'} 2025-10-10T01:50:17.9455120Z 2025-10-10T01:50:17.9457046Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:50:17.9457548Z config.json: 1.52kB [00:00, 8.87MB/s] 2025-10-10T01:50:18.0146451Z 2025-10-10T01:50:18.0148536Z configuration_bailing_moe_v2.py: 0.00B [00:00, ?B/s] 2025-10-10T01:50:18.0149340Z configuration_bailing_moe_v2.py: 3.16kB [00:00, 20.5MB/s] 2025-10-10T01:50:18.0238772Z A new version of the following files was downloaded from https://huggingface.co/inclusionAI/Ling-mini-2.0: 2025-10-10T01:50:18.0239424Z - configuration_bailing_moe_v2.py 2025-10-10T01:50:18.0240069Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:50:18.0691016Z You are using a model of type bailing_moe to instantiate a model of type . This is not supported for all configurations of models and can yield errors. 2025-10-10T01:50:24.7906463Z INFO 10-10 01:50:24 [model.py:551] Resolved architecture: BailingMoeV2ForCausalLM 2025-10-10T01:50:24.7906970Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:50:24.8154455Z INFO 10-10 01:50:24 [model.py:1545] Using max model len 32768 2025-10-10T01:50:24.8156647Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:50:24.8702218Z INFO 10-10 01:50:24 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:50:24.9246875Z 2025-10-10T01:50:24.9249748Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:50:24.9250107Z tokenizer_config.json: 4.59kB [00:00, 17.5MB/s] 2025-10-10T01:50:24.9951420Z 2025-10-10T01:50:25.0270688Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:50:25.0270996Z tokenizer.json: 7.66MB [00:00, 241MB/s] 2025-10-10T01:50:25.1329701Z 2025-10-10T01:50:25.1330740Z special_tokens_map.json: 0% 0.00/152 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:50:25.8045177Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] EngineCore failed to start. 2025-10-10T01:50:25.8045759Z 2025-10-10T01:50:25.8046212Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] Traceback (most recent call last): 2025-10-10T01:50:25.8046574Z 2025-10-10T01:50:25.8047256Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:25.8047841Z 2025-10-10T01:50:25.8048229Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:25.8048622Z 2025-10-10T01:50:25.8048972Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8049335Z 2025-10-10T01:50:25.8050000Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:25.8050477Z 2025-10-10T01:50:25.8050819Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:25.8051149Z 2025-10-10T01:50:25.8051640Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:25.8052165Z 2025-10-10T01:50:25.8052575Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:25.8052906Z 2025-10-10T01:50:25.8053192Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8053471Z 2025-10-10T01:50:25.8054090Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:25.8054558Z 2025-10-10T01:50:25.8054823Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] self._init_executor() 2025-10-10T01:50:25.8055082Z 2025-10-10T01:50:25.8055626Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:25.8056118Z 2025-10-10T01:50:25.8056594Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:25.8057195Z 2025-10-10T01:50:25.8058070Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:25.8058941Z 2025-10-10T01:50:25.8059630Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:25.8060061Z 2025-10-10T01:50:25.8060384Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8060675Z 2025-10-10T01:50:25.8061172Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:25.8061766Z 2025-10-10T01:50:25.8062036Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:50:25.8062304Z 2025-10-10T01:50:25.8062561Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8062898Z 2025-10-10T01:50:25.8063432Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:25.8063914Z 2025-10-10T01:50:25.8064209Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:25.8064505Z 2025-10-10T01:50:25.8064773Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8065048Z 2025-10-10T01:50:25.8065575Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:25.8066065Z 2025-10-10T01:50:25.8066369Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:50:25.8066731Z 2025-10-10T01:50:25.8067012Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8067281Z 2025-10-10T01:50:25.8067725Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:25.8068129Z 2025-10-10T01:50:25.8068469Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:25.8068796Z 2025-10-10T01:50:25.8069085Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8069361Z 2025-10-10T01:50:25.8069717Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:50:25.8070053Z 2025-10-10T01:50:25.8070442Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:50:25.8070783Z 2025-10-10T01:50:25.8071156Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:25.8071518Z 2025-10-10T01:50:25.8071860Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:50:25.8072199Z 2025-10-10T01:50:25.8072564Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:50:25.8072926Z 2025-10-10T01:50:25.8073304Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:25.8073669Z 2025-10-10T01:50:25.8074170Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:25.8074629Z 2025-10-10T01:50:25.8074985Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:25.8075317Z 2025-10-10T01:50:25.8075846Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:25.8076370Z 2025-10-10T01:50:25.8076739Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:25.8077097Z 2025-10-10T01:50:25.8077702Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:25.8078233Z 2025-10-10T01:50:25.8078527Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:50:25.8078826Z 2025-10-10T01:50:25.8079619Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:25.8080194Z 2025-10-10T01:50:25.8080564Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:25.8080917Z 2025-10-10T01:50:25.8081176Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8081433Z 2025-10-10T01:50:25.8082051Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:25.8082562Z 2025-10-10T01:50:25.8082862Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:50:25.8083158Z 2025-10-10T01:50:25.8083434Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8083702Z 2025-10-10T01:50:25.8084299Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:25.8084853Z 2025-10-10T01:50:25.8085130Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:50:25.8085437Z 2025-10-10T01:50:25.8085767Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8086042Z 2025-10-10T01:50:25.8086620Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:25.8087151Z 2025-10-10T01:50:25.8087465Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:25.8087775Z 2025-10-10T01:50:25.8088060Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8088331Z 2025-10-10T01:50:25.8088858Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:25.8089348Z 2025-10-10T01:50:25.8089637Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:50:25.8089920Z 2025-10-10T01:50:25.8090184Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8090457Z 2025-10-10T01:50:25.8090974Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:25.8091456Z 2025-10-10T01:50:25.8091810Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:25.8092120Z 2025-10-10T01:50:25.8092349Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:50:25.8092596Z 2025-10-10T01:50:25.8093123Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:25.8093573Z 2025-10-10T01:50:25.8093824Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] raise RuntimeError( 2025-10-10T01:50:25.8094077Z 2025-10-10T01:50:25.8094649Z (EngineCore_DP0 pid=7185) ERROR 10-10 01:50:25 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:25.8095363Z (EngineCore_DP0 pid=7185) Process EngineCore_DP0: 2025-10-10T01:50:25.8095767Z (EngineCore_DP0 pid=7185) Traceback (most recent call last): 2025-10-10T01:50:25.8096572Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:50:25.8097193Z (EngineCore_DP0 pid=7185) self.run() 2025-10-10T01:50:25.8097723Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:50:25.8098290Z (EngineCore_DP0 pid=7185) self._target(*self._args, **self._kwargs) 2025-10-10T01:50:25.8098961Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:50:25.8099508Z (EngineCore_DP0 pid=7185) raise e 2025-10-10T01:50:25.8100094Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:25.8100729Z (EngineCore_DP0 pid=7185) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:25.8101181Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8101877Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:25.8102515Z (EngineCore_DP0 pid=7185) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:25.8103177Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:25.8103796Z (EngineCore_DP0 pid=7185) self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:25.8104246Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8104896Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:25.8105471Z (EngineCore_DP0 pid=7185) self._init_executor() 2025-10-10T01:50:25.8106143Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:25.8106847Z (EngineCore_DP0 pid=7185) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:25.8107575Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:25.8108289Z (EngineCore_DP0 pid=7185) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:25.8108777Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8109497Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:25.8110083Z (EngineCore_DP0 pid=7185) return func(*args, **kwargs) 2025-10-10T01:50:25.8110530Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8111179Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:25.8111802Z (EngineCore_DP0 pid=7185) worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:25.8112227Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8112893Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:25.8113547Z (EngineCore_DP0 pid=7185) module = importlib.import_module(module_name) 2025-10-10T01:50:25.8114009Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8114592Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:25.8115275Z (EngineCore_DP0 pid=7185) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:25.8115767Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8116263Z (EngineCore_DP0 pid=7185) File "", line 1387, in _gcd_import 2025-10-10T01:50:25.8116818Z (EngineCore_DP0 pid=7185) File "", line 1360, in _find_and_load 2025-10-10T01:50:25.8117405Z (EngineCore_DP0 pid=7185) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:25.8117982Z (EngineCore_DP0 pid=7185) File "", line 935, in _load_unlocked 2025-10-10T01:50:25.8118561Z (EngineCore_DP0 pid=7185) File "", line 999, in exec_module 2025-10-10T01:50:25.8119276Z (EngineCore_DP0 pid=7185) File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:25.8120026Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:25.8120712Z (EngineCore_DP0 pid=7185) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:25.8121446Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:25.8122171Z (EngineCore_DP0 pid=7185) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:25.8122957Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:25.8123635Z (EngineCore_DP0 pid=7185) class FlashAttentionMetadataBuilder( 2025-10-10T01:50:25.8124430Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:25.8125243Z (EngineCore_DP0 pid=7185) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:25.8125720Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8126412Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:25.8127161Z (EngineCore_DP0 pid=7185) if not is_fa_version_supported(fa_version): 2025-10-10T01:50:25.8127601Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8128380Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:25.8129077Z (EngineCore_DP0 pid=7185) return _is_fa2_supported(device)[0] 2025-10-10T01:50:25.8129493Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8130198Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:25.8130929Z (EngineCore_DP0 pid=7185) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:25.8131382Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8132044Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:25.8132726Z (EngineCore_DP0 pid=7185) prop = get_device_properties(device) 2025-10-10T01:50:25.8133145Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:25.8133815Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:25.8134481Z (EngineCore_DP0 pid=7185) _lazy_init() # will define _get_device_properties 2025-10-10T01:50:25.8134874Z (EngineCore_DP0 pid=7185) ^^^^^^^^^^^^ 2025-10-10T01:50:25.8135450Z (EngineCore_DP0 pid=7185) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:25.8136014Z (EngineCore_DP0 pid=7185) raise RuntimeError( 2025-10-10T01:50:25.8136699Z (EngineCore_DP0 pid=7185) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:26.2188973Z FAILED 2025-10-10T01:50:26.2320099Z models/test_initialization.py::test_can_initialize_large_subset[CwmForCausalLM] Fork a new process to run a test 7189 2025-10-10T01:50:26.2330610Z Fork a new process to run a test 0 2025-10-10T01:50:26.2331830Z Model is not available online 2025-10-10T01:50:26.5419413Z PASSED 2025-10-10T01:50:26.5553301Z models/test_initialization.py::test_can_initialize_large_subset[TeleFLMForCausalLM] Fork a new process to run a test 7190 2025-10-10T01:50:26.5565785Z Fork a new process to run a test 0 2025-10-10T01:50:26.5840265Z INFO 10-10 01:50:26 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='TeleFLMForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'CofeAI/FLM-2-52B-Instruct-2407'} 2025-10-10T01:50:26.7353562Z 2025-10-10T01:50:26.7354952Z config.json: 0% 0.00/940 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:50:35.1020246Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] EngineCore failed to start. 2025-10-10T01:50:35.1020724Z 2025-10-10T01:50:35.1021147Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] Traceback (most recent call last): 2025-10-10T01:50:35.1021523Z 2025-10-10T01:50:35.1022458Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:35.1023062Z 2025-10-10T01:50:35.1023452Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:35.1023852Z 2025-10-10T01:50:35.1024192Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1024542Z 2025-10-10T01:50:35.1025194Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:35.1025942Z 2025-10-10T01:50:35.1026514Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:35.1026932Z 2025-10-10T01:50:35.1027662Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:35.1028191Z 2025-10-10T01:50:35.1028505Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:35.1028813Z 2025-10-10T01:50:35.1029144Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1029665Z 2025-10-10T01:50:35.1030633Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:35.1031522Z 2025-10-10T01:50:35.1032014Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] self._init_executor() 2025-10-10T01:50:35.1032488Z 2025-10-10T01:50:35.1033214Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:35.1033864Z 2025-10-10T01:50:35.1034223Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:35.1034549Z 2025-10-10T01:50:35.1035097Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:35.1035732Z 2025-10-10T01:50:35.1036084Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:35.1036421Z 2025-10-10T01:50:35.1036781Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1037070Z 2025-10-10T01:50:35.1037567Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:35.1038020Z 2025-10-10T01:50:35.1038284Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:50:35.1038549Z 2025-10-10T01:50:35.1038803Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1039210Z 2025-10-10T01:50:35.1039736Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:35.1040212Z 2025-10-10T01:50:35.1040503Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:35.1040838Z 2025-10-10T01:50:35.1041106Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1041389Z 2025-10-10T01:50:35.1041914Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:35.1042397Z 2025-10-10T01:50:35.1042694Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:50:35.1042996Z 2025-10-10T01:50:35.1043268Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1043541Z 2025-10-10T01:50:35.1043976Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:35.1044387Z 2025-10-10T01:50:35.1044775Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:35.1045107Z 2025-10-10T01:50:35.1045394Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1045666Z 2025-10-10T01:50:35.1046007Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:50:35.1046331Z 2025-10-10T01:50:35.1046673Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:50:35.1047008Z 2025-10-10T01:50:35.1047392Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:35.1047767Z 2025-10-10T01:50:35.1048119Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:50:35.1048456Z 2025-10-10T01:50:35.1048813Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:50:35.1049162Z 2025-10-10T01:50:35.1049527Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:35.1049904Z 2025-10-10T01:50:35.1050402Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:35.1050911Z 2025-10-10T01:50:35.1051293Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:35.1051638Z 2025-10-10T01:50:35.1052164Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:35.1052638Z 2025-10-10T01:50:35.1053015Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:35.1053374Z 2025-10-10T01:50:35.1053923Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:35.1054424Z 2025-10-10T01:50:35.1054718Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:50:35.1055008Z 2025-10-10T01:50:35.1055630Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:35.1056241Z 2025-10-10T01:50:35.1056604Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:35.1056956Z 2025-10-10T01:50:35.1057206Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1057477Z 2025-10-10T01:50:35.1058030Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:35.1058547Z 2025-10-10T01:50:35.1058841Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:50:35.1059130Z 2025-10-10T01:50:35.1059497Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1059768Z 2025-10-10T01:50:35.1060369Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:35.1060927Z 2025-10-10T01:50:35.1061213Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:50:35.1061489Z 2025-10-10T01:50:35.1061748Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1062011Z 2025-10-10T01:50:35.1062585Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:35.1063112Z 2025-10-10T01:50:35.1063431Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:35.1063741Z 2025-10-10T01:50:35.1064009Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1064281Z 2025-10-10T01:50:35.1064796Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:35.1065274Z 2025-10-10T01:50:35.1065607Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:50:35.1065887Z 2025-10-10T01:50:35.1066149Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1066415Z 2025-10-10T01:50:35.1066975Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:35.1067461Z 2025-10-10T01:50:35.1067770Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:35.1068067Z 2025-10-10T01:50:35.1068285Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:50:35.1068527Z 2025-10-10T01:50:35.1069001Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:35.1069449Z 2025-10-10T01:50:35.1069689Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] raise RuntimeError( 2025-10-10T01:50:35.1069947Z 2025-10-10T01:50:35.1070549Z (EngineCore_DP0 pid=7268) ERROR 10-10 01:50:35 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:35.1071238Z (EngineCore_DP0 pid=7268) Process EngineCore_DP0: 2025-10-10T01:50:35.1071661Z (EngineCore_DP0 pid=7268) Traceback (most recent call last): 2025-10-10T01:50:35.1072259Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:50:35.1072760Z (EngineCore_DP0 pid=7268) self.run() 2025-10-10T01:50:35.1073280Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:50:35.1073835Z (EngineCore_DP0 pid=7268) self._target(*self._args, **self._kwargs) 2025-10-10T01:50:35.1074530Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:50:35.1075131Z (EngineCore_DP0 pid=7268) raise e 2025-10-10T01:50:35.1075717Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:35.1076349Z (EngineCore_DP0 pid=7268) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:35.1076802Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1077418Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:35.1078055Z (EngineCore_DP0 pid=7268) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:35.1078713Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:35.1079380Z (EngineCore_DP0 pid=7268) self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:35.1079836Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1080481Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:35.1081065Z (EngineCore_DP0 pid=7268) self._init_executor() 2025-10-10T01:50:35.1081723Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:35.1082448Z (EngineCore_DP0 pid=7268) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:35.1083225Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:35.1083945Z (EngineCore_DP0 pid=7268) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:35.1084438Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1085076Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:35.1085669Z (EngineCore_DP0 pid=7268) return func(*args, **kwargs) 2025-10-10T01:50:35.1086060Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1086696Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:35.1087327Z (EngineCore_DP0 pid=7268) worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:35.1087799Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1088456Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:35.1089102Z (EngineCore_DP0 pid=7268) module = importlib.import_module(module_name) 2025-10-10T01:50:35.1089546Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1090120Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:35.1090750Z (EngineCore_DP0 pid=7268) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:35.1091232Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1091723Z (EngineCore_DP0 pid=7268) File "", line 1387, in _gcd_import 2025-10-10T01:50:35.1092320Z (EngineCore_DP0 pid=7268) File "", line 1360, in _find_and_load 2025-10-10T01:50:35.1092904Z (EngineCore_DP0 pid=7268) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:35.1093484Z (EngineCore_DP0 pid=7268) File "", line 935, in _load_unlocked 2025-10-10T01:50:35.1094080Z (EngineCore_DP0 pid=7268) File "", line 999, in exec_module 2025-10-10T01:50:35.1094684Z (EngineCore_DP0 pid=7268) File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:35.1095419Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:35.1096287Z (EngineCore_DP0 pid=7268) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:35.1097022Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:35.1097735Z (EngineCore_DP0 pid=7268) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:35.1098500Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:35.1099167Z (EngineCore_DP0 pid=7268) class FlashAttentionMetadataBuilder( 2025-10-10T01:50:35.1100058Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:35.1100870Z (EngineCore_DP0 pid=7268) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:35.1101416Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1102119Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:35.1102793Z (EngineCore_DP0 pid=7268) if not is_fa_version_supported(fa_version): 2025-10-10T01:50:35.1103231Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1103986Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:35.1104680Z (EngineCore_DP0 pid=7268) return _is_fa2_supported(device)[0] 2025-10-10T01:50:35.1105091Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1105798Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:35.1106580Z (EngineCore_DP0 pid=7268) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:35.1107034Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1107700Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:35.1108330Z (EngineCore_DP0 pid=7268) prop = get_device_properties(device) 2025-10-10T01:50:35.1108736Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:35.1109388Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:35.1110067Z (EngineCore_DP0 pid=7268) _lazy_init() # will define _get_device_properties 2025-10-10T01:50:35.1110526Z (EngineCore_DP0 pid=7268) ^^^^^^^^^^^^ 2025-10-10T01:50:35.1111110Z (EngineCore_DP0 pid=7268) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:35.1111661Z (EngineCore_DP0 pid=7268) raise RuntimeError( 2025-10-10T01:50:35.1112340Z (EngineCore_DP0 pid=7268) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:35.5126301Z FAILED 2025-10-10T01:50:35.5257484Z models/test_initialization.py::test_can_initialize_large_subset[GteNewModel] Fork a new process to run a test 7272 2025-10-10T01:50:35.5268710Z Fork a new process to run a test 0 2025-10-10T01:50:35.5543257Z INFO 10-10 01:50:35 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GteNewModel', exist_overrides={'architectures': ['GteNewModel']}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Alibaba-NLP/gte-base-en-v1.5'} 2025-10-10T01:50:35.7748417Z 2025-10-10T01:50:35.7750628Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:50:35.7751054Z config.json: 1.35kB [00:00, 7.04MB/s] 2025-10-10T01:50:35.8368534Z 2025-10-10T01:50:35.8370778Z configuration.py: 0.00B [00:00, ?B/s] 2025-10-10T01:50:35.8371207Z configuration.py: 7.13kB [00:00, 37.2MB/s] 2025-10-10T01:50:35.8463006Z A new version of the following files was downloaded from https://huggingface.co/Alibaba-NLP/new-impl: 2025-10-10T01:50:35.8463759Z - configuration.py 2025-10-10T01:50:35.8464364Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:50:35.8783850Z INFO 10-10 01:50:35 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:50:36.0642521Z 2025-10-10T01:50:36.0643089Z sentence_bert_config.json: 0% 0.00/54.0 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:50:43.5691692Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] EngineCore failed to start. 2025-10-10T01:50:43.5692261Z 2025-10-10T01:50:43.5692678Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] Traceback (most recent call last): 2025-10-10T01:50:43.5693082Z 2025-10-10T01:50:43.5693751Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:43.5694380Z 2025-10-10T01:50:43.5695068Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:43.5695488Z 2025-10-10T01:50:43.5695783Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5696287Z 2025-10-10T01:50:43.5696850Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:43.5697306Z 2025-10-10T01:50:43.5697649Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:43.5697979Z 2025-10-10T01:50:43.5698475Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:43.5698923Z 2025-10-10T01:50:43.5699250Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:43.5699555Z 2025-10-10T01:50:43.5699842Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5700125Z 2025-10-10T01:50:43.5700625Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:43.5701103Z 2025-10-10T01:50:43.5701355Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] self._init_executor() 2025-10-10T01:50:43.5701747Z 2025-10-10T01:50:43.5702307Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:43.5703093Z 2025-10-10T01:50:43.5703597Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:43.5704078Z 2025-10-10T01:50:43.5704946Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:43.5705664Z 2025-10-10T01:50:43.5706042Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:43.5706380Z 2025-10-10T01:50:43.5706682Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5706961Z 2025-10-10T01:50:43.5707449Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:43.5708051Z 2025-10-10T01:50:43.5708332Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:50:43.5708627Z 2025-10-10T01:50:43.5708883Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5709163Z 2025-10-10T01:50:43.5709685Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:43.5710168Z 2025-10-10T01:50:43.5710466Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:43.5710752Z 2025-10-10T01:50:43.5711022Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5711291Z 2025-10-10T01:50:43.5711935Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:43.5712430Z 2025-10-10T01:50:43.5712739Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:50:43.5713050Z 2025-10-10T01:50:43.5713344Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5713619Z 2025-10-10T01:50:43.5714076Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:43.5714488Z 2025-10-10T01:50:43.5714828Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:43.5715164Z 2025-10-10T01:50:43.5715454Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5715744Z 2025-10-10T01:50:43.5716083Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:50:43.5716426Z 2025-10-10T01:50:43.5716772Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:50:43.5717104Z 2025-10-10T01:50:43.5717481Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:43.5717885Z 2025-10-10T01:50:43.5718233Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:50:43.5718573Z 2025-10-10T01:50:43.5719100Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:50:43.5719459Z 2025-10-10T01:50:43.5719838Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:43.5720203Z 2025-10-10T01:50:43.5720696Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:43.5721164Z 2025-10-10T01:50:43.5721516Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:43.5721871Z 2025-10-10T01:50:43.5722392Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:43.5722934Z 2025-10-10T01:50:43.5723326Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:43.5723692Z 2025-10-10T01:50:43.5724252Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:43.5724751Z 2025-10-10T01:50:43.5725061Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:50:43.5725354Z 2025-10-10T01:50:43.5725981Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:43.5726552Z 2025-10-10T01:50:43.5726971Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:43.5727323Z 2025-10-10T01:50:43.5727578Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5727855Z 2025-10-10T01:50:43.5728413Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:43.5728932Z 2025-10-10T01:50:43.5729227Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:50:43.5729530Z 2025-10-10T01:50:43.5729796Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5730072Z 2025-10-10T01:50:43.5730672Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:43.5731217Z 2025-10-10T01:50:43.5731499Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:50:43.5731790Z 2025-10-10T01:50:43.5732051Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5732313Z 2025-10-10T01:50:43.5732890Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:43.5733461Z 2025-10-10T01:50:43.5733783Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:43.5734127Z 2025-10-10T01:50:43.5734420Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5734717Z 2025-10-10T01:50:43.5735250Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:43.5735737Z 2025-10-10T01:50:43.5736017Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:50:43.5736302Z 2025-10-10T01:50:43.5736559Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5736828Z 2025-10-10T01:50:43.5737351Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:43.5737882Z 2025-10-10T01:50:43.5738212Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:43.5738514Z 2025-10-10T01:50:43.5738744Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:50:43.5738982Z 2025-10-10T01:50:43.5739459Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:43.5739910Z 2025-10-10T01:50:43.5740150Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] raise RuntimeError( 2025-10-10T01:50:43.5740416Z 2025-10-10T01:50:43.5740976Z (EngineCore_DP0 pid=7331) ERROR 10-10 01:50:43 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:43.5741705Z (EngineCore_DP0 pid=7331) Process EngineCore_DP0: 2025-10-10T01:50:43.5742145Z (EngineCore_DP0 pid=7331) Traceback (most recent call last): 2025-10-10T01:50:43.5742772Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:50:43.5743304Z (EngineCore_DP0 pid=7331) self.run() 2025-10-10T01:50:43.5743825Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:50:43.5744399Z (EngineCore_DP0 pid=7331) self._target(*self._args, **self._kwargs) 2025-10-10T01:50:43.5745075Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:50:43.5745643Z (EngineCore_DP0 pid=7331) raise e 2025-10-10T01:50:43.5746238Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:43.5746876Z (EngineCore_DP0 pid=7331) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:43.5747319Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5747939Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:43.5748577Z (EngineCore_DP0 pid=7331) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:43.5749293Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:43.5749911Z (EngineCore_DP0 pid=7331) self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:43.5750421Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5751095Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:43.5751678Z (EngineCore_DP0 pid=7331) self._init_executor() 2025-10-10T01:50:43.5752336Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:43.5753024Z (EngineCore_DP0 pid=7331) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:43.5753737Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:43.5754443Z (EngineCore_DP0 pid=7331) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:43.5754996Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5755640Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:43.5756221Z (EngineCore_DP0 pid=7331) return func(*args, **kwargs) 2025-10-10T01:50:43.5756598Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5757229Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:43.5757856Z (EngineCore_DP0 pid=7331) worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:43.5758277Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5758943Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:43.5759734Z (EngineCore_DP0 pid=7331) module = importlib.import_module(module_name) 2025-10-10T01:50:43.5760191Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5760769Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:43.5761394Z (EngineCore_DP0 pid=7331) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:43.5761894Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5762401Z (EngineCore_DP0 pid=7331) File "", line 1387, in _gcd_import 2025-10-10T01:50:43.5762949Z (EngineCore_DP0 pid=7331) File "", line 1360, in _find_and_load 2025-10-10T01:50:43.5763538Z (EngineCore_DP0 pid=7331) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:43.5764117Z (EngineCore_DP0 pid=7331) File "", line 935, in _load_unlocked 2025-10-10T01:50:43.5764696Z (EngineCore_DP0 pid=7331) File "", line 999, in exec_module 2025-10-10T01:50:43.5765297Z (EngineCore_DP0 pid=7331) File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:43.5766029Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:43.5766777Z (EngineCore_DP0 pid=7331) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:43.5767500Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:43.5768275Z (EngineCore_DP0 pid=7331) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:43.5769058Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:43.5769724Z (EngineCore_DP0 pid=7331) class FlashAttentionMetadataBuilder( 2025-10-10T01:50:43.5770514Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:43.5771327Z (EngineCore_DP0 pid=7331) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:43.5771820Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5772515Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:43.5773245Z (EngineCore_DP0 pid=7331) if not is_fa_version_supported(fa_version): 2025-10-10T01:50:43.5773687Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5774432Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:43.5775121Z (EngineCore_DP0 pid=7331) return _is_fa2_supported(device)[0] 2025-10-10T01:50:43.5775679Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5776411Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:43.5777118Z (EngineCore_DP0 pid=7331) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:43.5777622Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5778317Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:43.5778957Z (EngineCore_DP0 pid=7331) prop = get_device_properties(device) 2025-10-10T01:50:43.5779375Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:43.5780021Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:43.5780701Z (EngineCore_DP0 pid=7331) _lazy_init() # will define _get_device_properties 2025-10-10T01:50:43.5781097Z (EngineCore_DP0 pid=7331) ^^^^^^^^^^^^ 2025-10-10T01:50:43.5781671Z (EngineCore_DP0 pid=7331) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:43.5782234Z (EngineCore_DP0 pid=7331) raise RuntimeError( 2025-10-10T01:50:43.5782920Z (EngineCore_DP0 pid=7331) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:43.9773079Z FAILED 2025-10-10T01:50:43.9904352Z models/test_initialization.py::test_can_initialize_large_subset[DotsOCRForCausalLM] Fork a new process to run a test 7335 2025-10-10T01:50:43.9915456Z Fork a new process to run a test 0 2025-10-10T01:50:44.0191752Z INFO 10-10 01:50:44 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DotsOCRForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'rednote-hilab/dots.ocr'} 2025-10-10T01:50:44.0951960Z 2025-10-10T01:50:44.0953964Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:50:44.0954289Z config.json: 1.47kB [00:00, 8.72MB/s] 2025-10-10T01:50:44.1939347Z 2025-10-10T01:50:44.1940499Z configuration_dots.py: 0.00B [00:00, ?B/s] 2025-10-10T01:50:44.1941010Z configuration_dots.py: 3.04kB [00:00, 36.6MB/s] 2025-10-10T01:50:44.2030799Z A new version of the following files was downloaded from https://huggingface.co/rednote-hilab/dots.ocr: 2025-10-10T01:50:44.2031330Z - configuration_dots.py 2025-10-10T01:50:44.2031943Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:50:44.4402823Z 2025-10-10T01:50:44.4403363Z preprocessor_config.json: 0% 0.00/432 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:50:52.6166789Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] EngineCore failed to start. 2025-10-10T01:50:52.6167537Z 2025-10-10T01:50:52.6167951Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] Traceback (most recent call last): 2025-10-10T01:50:52.6168338Z 2025-10-10T01:50:52.6169051Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:52.6169647Z 2025-10-10T01:50:52.6170040Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:52.6170430Z 2025-10-10T01:50:52.6170780Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6171311Z 2025-10-10T01:50:52.6171934Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:52.6172502Z 2025-10-10T01:50:52.6173045Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:52.6173464Z 2025-10-10T01:50:52.6174051Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:52.6174725Z 2025-10-10T01:50:52.6175333Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:52.6175857Z 2025-10-10T01:50:52.6176425Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6176786Z 2025-10-10T01:50:52.6177693Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:52.6178208Z 2025-10-10T01:50:52.6178507Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] self._init_executor() 2025-10-10T01:50:52.6178786Z 2025-10-10T01:50:52.6179338Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:52.6179836Z 2025-10-10T01:50:52.6180156Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:52.6180481Z 2025-10-10T01:50:52.6181017Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:52.6181644Z 2025-10-10T01:50:52.6182070Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:52.6182426Z 2025-10-10T01:50:52.6182724Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6183012Z 2025-10-10T01:50:52.6183504Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:52.6183953Z 2025-10-10T01:50:52.6184223Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:50:52.6184490Z 2025-10-10T01:50:52.6184745Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6185001Z 2025-10-10T01:50:52.6185517Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:52.6186031Z 2025-10-10T01:50:52.6186321Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:52.6186611Z 2025-10-10T01:50:52.6186873Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6187144Z 2025-10-10T01:50:52.6187661Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:52.6188143Z 2025-10-10T01:50:52.6188442Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:50:52.6188747Z 2025-10-10T01:50:52.6189016Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6189289Z 2025-10-10T01:50:52.6189786Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:52.6190222Z 2025-10-10T01:50:52.6197372Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:52.6197800Z 2025-10-10T01:50:52.6198172Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6198482Z 2025-10-10T01:50:52.6199151Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:50:52.6199547Z 2025-10-10T01:50:52.6199931Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:50:52.6200281Z 2025-10-10T01:50:52.6200671Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:52.6201034Z 2025-10-10T01:50:52.6201391Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:50:52.6201725Z 2025-10-10T01:50:52.6202088Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:50:52.6202447Z 2025-10-10T01:50:52.6202820Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:52.6203313Z 2025-10-10T01:50:52.6203902Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:52.6204394Z 2025-10-10T01:50:52.6204761Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:52.6205113Z 2025-10-10T01:50:52.6205658Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:52.6206152Z 2025-10-10T01:50:52.6206536Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:52.6206897Z 2025-10-10T01:50:52.6207455Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:52.6207961Z 2025-10-10T01:50:52.6208261Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:50:52.6208621Z 2025-10-10T01:50:52.6209255Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:52.6210001Z 2025-10-10T01:50:52.6210400Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:52.6210769Z 2025-10-10T01:50:52.6211031Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6211306Z 2025-10-10T01:50:52.6211881Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:52.6212404Z 2025-10-10T01:50:52.6212805Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:50:52.6213134Z 2025-10-10T01:50:52.6213419Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6213693Z 2025-10-10T01:50:52.6214316Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:52.6214868Z 2025-10-10T01:50:52.6215157Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:50:52.6215454Z 2025-10-10T01:50:52.6215720Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6215985Z 2025-10-10T01:50:52.6216567Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:52.6217102Z 2025-10-10T01:50:52.6217419Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:52.6217734Z 2025-10-10T01:50:52.6218004Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6218281Z 2025-10-10T01:50:52.6218801Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:52.6219337Z 2025-10-10T01:50:52.6219620Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:50:52.6219941Z 2025-10-10T01:50:52.6220222Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6220494Z 2025-10-10T01:50:52.6221036Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:52.6221516Z 2025-10-10T01:50:52.6221829Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:52.6222131Z 2025-10-10T01:50:52.6222356Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:50:52.6222608Z 2025-10-10T01:50:52.6223097Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:52.6223601Z 2025-10-10T01:50:52.6223851Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] raise RuntimeError( 2025-10-10T01:50:52.6224112Z 2025-10-10T01:50:52.6224680Z (EngineCore_DP0 pid=7393) ERROR 10-10 01:50:52 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:52.6225374Z (EngineCore_DP0 pid=7393) Process EngineCore_DP0: 2025-10-10T01:50:52.6225790Z (EngineCore_DP0 pid=7393) Traceback (most recent call last): 2025-10-10T01:50:52.6226398Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:50:52.6226917Z (EngineCore_DP0 pid=7393) self.run() 2025-10-10T01:50:52.6227443Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:50:52.6228012Z (EngineCore_DP0 pid=7393) self._target(*self._args, **self._kwargs) 2025-10-10T01:50:52.6228718Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:50:52.6229270Z (EngineCore_DP0 pid=7393) raise e 2025-10-10T01:50:52.6229857Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:52.6230483Z (EngineCore_DP0 pid=7393) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:52.6230926Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6231549Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:52.6232192Z (EngineCore_DP0 pid=7393) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:52.6232863Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:52.6233494Z (EngineCore_DP0 pid=7393) self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:52.6233949Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6234606Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:52.6235242Z (EngineCore_DP0 pid=7393) self._init_executor() 2025-10-10T01:50:52.6235906Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:52.6236640Z (EngineCore_DP0 pid=7393) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:52.6237380Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:52.6238096Z (EngineCore_DP0 pid=7393) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:52.6238597Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6239305Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:52.6239895Z (EngineCore_DP0 pid=7393) return func(*args, **kwargs) 2025-10-10T01:50:52.6240277Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6240917Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:52.6241596Z (EngineCore_DP0 pid=7393) worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:52.6242025Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6242692Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:52.6243350Z (EngineCore_DP0 pid=7393) module = importlib.import_module(module_name) 2025-10-10T01:50:52.6243805Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6244387Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:52.6245018Z (EngineCore_DP0 pid=7393) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:52.6245513Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6246044Z (EngineCore_DP0 pid=7393) File "", line 1387, in _gcd_import 2025-10-10T01:50:52.6246624Z (EngineCore_DP0 pid=7393) File "", line 1360, in _find_and_load 2025-10-10T01:50:52.6247218Z (EngineCore_DP0 pid=7393) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:52.6247801Z (EngineCore_DP0 pid=7393) File "", line 935, in _load_unlocked 2025-10-10T01:50:52.6248375Z (EngineCore_DP0 pid=7393) File "", line 999, in exec_module 2025-10-10T01:50:52.6248969Z (EngineCore_DP0 pid=7393) File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:52.6249701Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:52.6250378Z (EngineCore_DP0 pid=7393) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:52.6251095Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:52.6251812Z (EngineCore_DP0 pid=7393) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:52.6252592Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:52.6253303Z (EngineCore_DP0 pid=7393) class FlashAttentionMetadataBuilder( 2025-10-10T01:50:52.6254148Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:52.6254966Z (EngineCore_DP0 pid=7393) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:52.6255446Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6256136Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:52.6256816Z (EngineCore_DP0 pid=7393) if not is_fa_version_supported(fa_version): 2025-10-10T01:50:52.6257254Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6257998Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:52.6258693Z (EngineCore_DP0 pid=7393) return _is_fa2_supported(device)[0] 2025-10-10T01:50:52.6259169Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6259876Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:52.6260592Z (EngineCore_DP0 pid=7393) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:52.6261045Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6261717Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:52.6262362Z (EngineCore_DP0 pid=7393) prop = get_device_properties(device) 2025-10-10T01:50:52.6262772Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:52.6263482Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:52.6264147Z (EngineCore_DP0 pid=7393) _lazy_init() # will define _get_device_properties 2025-10-10T01:50:52.6264549Z (EngineCore_DP0 pid=7393) ^^^^^^^^^^^^ 2025-10-10T01:50:52.6265142Z (EngineCore_DP0 pid=7393) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:52.6265699Z (EngineCore_DP0 pid=7393) raise RuntimeError( 2025-10-10T01:50:52.6266386Z (EngineCore_DP0 pid=7393) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:53.0364029Z FAILED 2025-10-10T01:50:53.0494558Z models/test_initialization.py::test_can_initialize_large_subset[RobertaForSequenceClassification] Fork a new process to run a test 7397 2025-10-10T01:50:53.0507125Z Fork a new process to run a test 0 2025-10-10T01:50:53.0790142Z INFO 10-10 01:50:53 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='RobertaForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'cross-encoder/quora-roberta-base'} 2025-10-10T01:50:53.3793387Z 2025-10-10T01:50:53.3795271Z config.json: 0% 0.00/608 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:50:54.5350714Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] EngineCore failed to start. 2025-10-10T01:50:54.5351446Z 2025-10-10T01:50:54.5351849Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] Traceback (most recent call last): 2025-10-10T01:50:54.5352206Z 2025-10-10T01:50:54.5352886Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:54.5353505Z 2025-10-10T01:50:54.5353888Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:54.5354285Z 2025-10-10T01:50:54.5354632Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5354990Z 2025-10-10T01:50:54.5355605Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:54.5356201Z 2025-10-10T01:50:54.5356535Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:54.5356848Z 2025-10-10T01:50:54.5357322Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:54.5357756Z 2025-10-10T01:50:54.5358065Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:54.5358380Z 2025-10-10T01:50:54.5358666Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5359133Z 2025-10-10T01:50:54.5360035Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:54.5360898Z 2025-10-10T01:50:54.5361317Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] self._init_executor() 2025-10-10T01:50:54.5361672Z 2025-10-10T01:50:54.5362644Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:54.5363353Z 2025-10-10T01:50:54.5363703Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:54.5364038Z 2025-10-10T01:50:54.5364586Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:54.5365081Z 2025-10-10T01:50:54.5365441Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:54.5365773Z 2025-10-10T01:50:54.5366069Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5366346Z 2025-10-10T01:50:54.5366858Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:54.5367330Z 2025-10-10T01:50:54.5367607Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:50:54.5367981Z 2025-10-10T01:50:54.5368231Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5368491Z 2025-10-10T01:50:54.5369111Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:54.5369600Z 2025-10-10T01:50:54.5369888Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:54.5370180Z 2025-10-10T01:50:54.5370441Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5370704Z 2025-10-10T01:50:54.5371232Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:54.5371714Z 2025-10-10T01:50:54.5372021Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:50:54.5372316Z 2025-10-10T01:50:54.5372593Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5372906Z 2025-10-10T01:50:54.5373345Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:54.5373784Z 2025-10-10T01:50:54.5374125Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:54.5374463Z 2025-10-10T01:50:54.5374746Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5375030Z 2025-10-10T01:50:54.5375376Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:50:54.5375708Z 2025-10-10T01:50:54.5376057Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:50:54.5376400Z 2025-10-10T01:50:54.5376821Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:54.5377181Z 2025-10-10T01:50:54.5377527Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:50:54.5377856Z 2025-10-10T01:50:54.5378222Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:50:54.5378572Z 2025-10-10T01:50:54.5378940Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:54.5379307Z 2025-10-10T01:50:54.5379807Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:54.5380278Z 2025-10-10T01:50:54.5380627Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:54.5380961Z 2025-10-10T01:50:54.5381474Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:54.5381955Z 2025-10-10T01:50:54.5382322Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:54.5382723Z 2025-10-10T01:50:54.5383285Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:54.5383794Z 2025-10-10T01:50:54.5384133Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:50:54.5384430Z 2025-10-10T01:50:54.5385076Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:54.5385654Z 2025-10-10T01:50:54.5386034Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:54.5386385Z 2025-10-10T01:50:54.5386642Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5386907Z 2025-10-10T01:50:54.5387467Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:54.5388026Z 2025-10-10T01:50:54.5388323Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:50:54.5388636Z 2025-10-10T01:50:54.5388907Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5389182Z 2025-10-10T01:50:54.5389777Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:54.5390324Z 2025-10-10T01:50:54.5390607Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:50:54.5390887Z 2025-10-10T01:50:54.5391149Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5391415Z 2025-10-10T01:50:54.5392037Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:54.5392563Z 2025-10-10T01:50:54.5392881Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:54.5393210Z 2025-10-10T01:50:54.5393484Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5393763Z 2025-10-10T01:50:54.5394295Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:54.5394773Z 2025-10-10T01:50:54.5395053Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:50:54.5395349Z 2025-10-10T01:50:54.5395608Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5395878Z 2025-10-10T01:50:54.5396668Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:54.5397163Z 2025-10-10T01:50:54.5397484Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:50:54.5397887Z 2025-10-10T01:50:54.5398123Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:50:54.5398359Z 2025-10-10T01:50:54.5398916Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:54.5399497Z 2025-10-10T01:50:54.5399757Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] raise RuntimeError( 2025-10-10T01:50:54.5400014Z 2025-10-10T01:50:54.5400581Z (EngineCore_DP0 pid=7405) ERROR 10-10 01:50:54 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:54.5401275Z (EngineCore_DP0 pid=7405) Process EngineCore_DP0: 2025-10-10T01:50:54.5401671Z (EngineCore_DP0 pid=7405) Traceback (most recent call last): 2025-10-10T01:50:54.5402281Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:50:54.5402795Z (EngineCore_DP0 pid=7405) self.run() 2025-10-10T01:50:54.5403314Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:50:54.5403947Z (EngineCore_DP0 pid=7405) self._target(*self._args, **self._kwargs) 2025-10-10T01:50:54.5404611Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:50:54.5405150Z (EngineCore_DP0 pid=7405) raise e 2025-10-10T01:50:54.5405745Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:50:54.5406374Z (EngineCore_DP0 pid=7405) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:50:54.5406810Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5407431Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:50:54.5408072Z (EngineCore_DP0 pid=7405) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:50:54.5408803Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:50:54.5409429Z (EngineCore_DP0 pid=7405) self.model_executor = executor_class(vllm_config) 2025-10-10T01:50:54.5409873Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5410519Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:50:54.5411105Z (EngineCore_DP0 pid=7405) self._init_executor() 2025-10-10T01:50:54.5411763Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:50:54.5412451Z (EngineCore_DP0 pid=7405) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:50:54.5413168Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:50:54.5413874Z (EngineCore_DP0 pid=7405) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:50:54.5414365Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5415005Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:50:54.5415639Z (EngineCore_DP0 pid=7405) return func(*args, **kwargs) 2025-10-10T01:50:54.5416014Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5416685Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:50:54.5417323Z (EngineCore_DP0 pid=7405) worker_class = resolve_obj_by_qualname( 2025-10-10T01:50:54.5417748Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5418410Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:50:54.5419055Z (EngineCore_DP0 pid=7405) module = importlib.import_module(module_name) 2025-10-10T01:50:54.5419507Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5420098Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:50:54.5420722Z (EngineCore_DP0 pid=7405) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:50:54.5421261Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5421751Z (EngineCore_DP0 pid=7405) File "", line 1387, in _gcd_import 2025-10-10T01:50:54.5422301Z (EngineCore_DP0 pid=7405) File "", line 1360, in _find_and_load 2025-10-10T01:50:54.5422887Z (EngineCore_DP0 pid=7405) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:50:54.5423462Z (EngineCore_DP0 pid=7405) File "", line 935, in _load_unlocked 2025-10-10T01:50:54.5424027Z (EngineCore_DP0 pid=7405) File "", line 999, in exec_module 2025-10-10T01:50:54.5424627Z (EngineCore_DP0 pid=7405) File "", line 488, in _call_with_frames_removed 2025-10-10T01:50:54.5425397Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:50:54.5426089Z (EngineCore_DP0 pid=7405) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:50:54.5426829Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:50:54.5427553Z (EngineCore_DP0 pid=7405) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:50:54.5428330Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:50:54.5428996Z (EngineCore_DP0 pid=7405) class FlashAttentionMetadataBuilder( 2025-10-10T01:50:54.5429786Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:50:54.5430609Z (EngineCore_DP0 pid=7405) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:50:54.5431092Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5431779Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:50:54.5432460Z (EngineCore_DP0 pid=7405) if not is_fa_version_supported(fa_version): 2025-10-10T01:50:54.5432966Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5433708Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:50:54.5434444Z (EngineCore_DP0 pid=7405) return _is_fa2_supported(device)[0] 2025-10-10T01:50:54.5434862Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5435576Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:50:54.5436282Z (EngineCore_DP0 pid=7405) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:50:54.5436734Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5437397Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:50:54.5438037Z (EngineCore_DP0 pid=7405) prop = get_device_properties(device) 2025-10-10T01:50:54.5438453Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:50:54.5439231Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:50:54.5439901Z (EngineCore_DP0 pid=7405) _lazy_init() # will define _get_device_properties 2025-10-10T01:50:54.5440297Z (EngineCore_DP0 pid=7405) ^^^^^^^^^^^^ 2025-10-10T01:50:54.5440874Z (EngineCore_DP0 pid=7405) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:50:54.5441437Z (EngineCore_DP0 pid=7405) raise RuntimeError( 2025-10-10T01:50:54.5442113Z (EngineCore_DP0 pid=7405) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:50:54.9492987Z FAILED 2025-10-10T01:50:54.9624137Z models/test_initialization.py::test_can_initialize_large_subset[BloomForCausalLM] Fork a new process to run a test 7409 2025-10-10T01:50:54.9635361Z Fork a new process to run a test 0 2025-10-10T01:50:54.9914000Z INFO 10-10 01:50:54 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BloomForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'bigscience/bloom-560m'} 2025-10-10T01:50:55.0910186Z 2025-10-10T01:50:55.0912045Z config.json: 0% 0.00/693 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:03.7043020Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:03.7043400Z 2025-10-10T01:51:03.7043788Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:03.7044182Z 2025-10-10T01:51:03.7044814Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:03.7045305Z 2025-10-10T01:51:03.7045639Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:03.7045973Z 2025-10-10T01:51:03.7046251Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7046533Z 2025-10-10T01:51:03.7047022Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:03.7047470Z 2025-10-10T01:51:03.7047793Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:03.7048325Z 2025-10-10T01:51:03.7048801Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:03.7049239Z 2025-10-10T01:51:03.7049700Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:03.7050127Z 2025-10-10T01:51:03.7050445Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7050725Z 2025-10-10T01:51:03.7051252Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:03.7051731Z 2025-10-10T01:51:03.7051978Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] self._init_executor() 2025-10-10T01:51:03.7052248Z 2025-10-10T01:51:03.7052892Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:03.7053789Z 2025-10-10T01:51:03.7054415Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:03.7054955Z 2025-10-10T01:51:03.7055536Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:03.7056284Z 2025-10-10T01:51:03.7056887Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:03.7057505Z 2025-10-10T01:51:03.7057867Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7058152Z 2025-10-10T01:51:03.7058704Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:03.7059176Z 2025-10-10T01:51:03.7059563Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:03.7059850Z 2025-10-10T01:51:03.7060122Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7060379Z 2025-10-10T01:51:03.7060901Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:03.7061374Z 2025-10-10T01:51:03.7061666Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:03.7061954Z 2025-10-10T01:51:03.7062233Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7062516Z 2025-10-10T01:51:03.7063069Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:03.7063563Z 2025-10-10T01:51:03.7063863Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:03.7064182Z 2025-10-10T01:51:03.7064454Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7064731Z 2025-10-10T01:51:03.7065174Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:03.7065669Z 2025-10-10T01:51:03.7066016Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:03.7066383Z 2025-10-10T01:51:03.7066691Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7066970Z 2025-10-10T01:51:03.7067314Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:03.7067642Z 2025-10-10T01:51:03.7067981Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:03.7071184Z 2025-10-10T01:51:03.7071616Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:03.7071989Z 2025-10-10T01:51:03.7072337Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:03.7072747Z 2025-10-10T01:51:03.7073126Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:03.7073482Z 2025-10-10T01:51:03.7073858Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:03.7074212Z 2025-10-10T01:51:03.7074732Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:03.7075219Z 2025-10-10T01:51:03.7075571Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:03.7075916Z 2025-10-10T01:51:03.7076433Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:03.7076919Z 2025-10-10T01:51:03.7077331Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:03.7077692Z 2025-10-10T01:51:03.7078242Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:03.7078749Z 2025-10-10T01:51:03.7079204Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:03.7079504Z 2025-10-10T01:51:03.7080138Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:03.7080706Z 2025-10-10T01:51:03.7081083Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:03.7081431Z 2025-10-10T01:51:03.7081688Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7081947Z 2025-10-10T01:51:03.7082505Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:03.7083019Z 2025-10-10T01:51:03.7083315Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:03.7083616Z 2025-10-10T01:51:03.7083887Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7084163Z 2025-10-10T01:51:03.7084810Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:03.7085377Z 2025-10-10T01:51:03.7085659Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:03.7085964Z 2025-10-10T01:51:03.7086221Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7086490Z 2025-10-10T01:51:03.7087069Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:03.7087702Z 2025-10-10T01:51:03.7088026Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:03.7088379Z 2025-10-10T01:51:03.7088663Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7088935Z 2025-10-10T01:51:03.7089465Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:03.7089946Z 2025-10-10T01:51:03.7090227Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:03.7090509Z 2025-10-10T01:51:03.7090770Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7091043Z 2025-10-10T01:51:03.7091554Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:03.7092042Z 2025-10-10T01:51:03.7092392Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:03.7092705Z 2025-10-10T01:51:03.7092937Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:03.7093185Z 2025-10-10T01:51:03.7093678Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:03.7094128Z 2025-10-10T01:51:03.7094383Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] raise RuntimeError( 2025-10-10T01:51:03.7094640Z 2025-10-10T01:51:03.7095203Z (EngineCore_DP0 pid=7487) ERROR 10-10 01:51:03 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:03.7095901Z (EngineCore_DP0 pid=7487) Process EngineCore_DP0: 2025-10-10T01:51:03.7096547Z (EngineCore_DP0 pid=7487) Traceback (most recent call last): 2025-10-10T01:51:03.7097151Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:03.7097664Z (EngineCore_DP0 pid=7487) self.run() 2025-10-10T01:51:03.7098192Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:03.7098758Z (EngineCore_DP0 pid=7487) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:03.7099417Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:03.7099957Z (EngineCore_DP0 pid=7487) raise e 2025-10-10T01:51:03.7100622Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:03.7101276Z (EngineCore_DP0 pid=7487) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:03.7101711Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7102330Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:03.7102961Z (EngineCore_DP0 pid=7487) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:03.7103723Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:03.7104339Z (EngineCore_DP0 pid=7487) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:03.7104792Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7105506Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:03.7106090Z (EngineCore_DP0 pid=7487) self._init_executor() 2025-10-10T01:51:03.7106743Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:03.7107422Z (EngineCore_DP0 pid=7487) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:03.7108158Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:03.7108858Z (EngineCore_DP0 pid=7487) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:03.7109354Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7110059Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:03.7110650Z (EngineCore_DP0 pid=7487) return func(*args, **kwargs) 2025-10-10T01:51:03.7111035Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7111662Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:03.7112289Z (EngineCore_DP0 pid=7487) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:03.7112715Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7113398Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:03.7114054Z (EngineCore_DP0 pid=7487) module = importlib.import_module(module_name) 2025-10-10T01:51:03.7114491Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7115071Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:03.7115690Z (EngineCore_DP0 pid=7487) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:03.7116190Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7116708Z (EngineCore_DP0 pid=7487) File "", line 1387, in _gcd_import 2025-10-10T01:51:03.7117256Z (EngineCore_DP0 pid=7487) File "", line 1360, in _find_and_load 2025-10-10T01:51:03.7117888Z (EngineCore_DP0 pid=7487) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:03.7118476Z (EngineCore_DP0 pid=7487) File "", line 935, in _load_unlocked 2025-10-10T01:51:03.7119141Z (EngineCore_DP0 pid=7487) File "", line 999, in exec_module 2025-10-10T01:51:03.7119752Z (EngineCore_DP0 pid=7487) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:03.7120496Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:03.7121229Z (EngineCore_DP0 pid=7487) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:03.7121972Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:03.7122759Z (EngineCore_DP0 pid=7487) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:03.7123551Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:03.7124221Z (EngineCore_DP0 pid=7487) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:03.7125017Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:03.7125836Z (EngineCore_DP0 pid=7487) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:03.7126320Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7127012Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:03.7127735Z (EngineCore_DP0 pid=7487) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:03.7128183Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7128932Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:03.7129641Z (EngineCore_DP0 pid=7487) return _is_fa2_supported(device)[0] 2025-10-10T01:51:03.7130058Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7130765Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:03.7131480Z (EngineCore_DP0 pid=7487) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:03.7131934Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7132599Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:03.7133235Z (EngineCore_DP0 pid=7487) prop = get_device_properties(device) 2025-10-10T01:51:03.7133653Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:03.7134319Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:03.7134979Z (EngineCore_DP0 pid=7487) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:03.7135371Z (EngineCore_DP0 pid=7487) ^^^^^^^^^^^^ 2025-10-10T01:51:03.7135994Z (EngineCore_DP0 pid=7487) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:03.7136562Z (EngineCore_DP0 pid=7487) raise RuntimeError( 2025-10-10T01:51:03.7137244Z (EngineCore_DP0 pid=7487) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:04.1302708Z FAILED 2025-10-10T01:51:04.1434068Z models/test_initialization.py::test_can_initialize_large_subset[KeyeForConditionalGeneration] Fork a new process to run a test 7491 2025-10-10T01:51:04.1444986Z Fork a new process to run a test 0 2025-10-10T01:51:04.1726501Z INFO 10-10 01:51:04 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='KeyeForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Kwai-Keye/Keye-VL-8B-Preview'} 2025-10-10T01:51:04.2857715Z 2025-10-10T01:51:04.2859637Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:51:04.2860067Z config.json: 1.87kB [00:00, 9.78MB/s] 2025-10-10T01:51:04.3576945Z 2025-10-10T01:51:04.3579610Z configuration_keye.py: 0.00B [00:00, ?B/s] 2025-10-10T01:51:04.3580152Z configuration_keye.py: 12.3kB [00:00, 48.2MB/s] 2025-10-10T01:51:04.3678927Z A new version of the following files was downloaded from https://huggingface.co/Kwai-Keye/Keye-VL-8B-Preview: 2025-10-10T01:51:04.3679608Z - configuration_keye.py 2025-10-10T01:51:04.3680239Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:51:04.5849110Z 2025-10-10T01:51:04.5849549Z preprocessor_config.json: 0% 0.00/380 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:12.9485342Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:12.9485803Z 2025-10-10T01:51:12.9486333Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:12.9486821Z 2025-10-10T01:51:12.9487677Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:12.9488335Z 2025-10-10T01:51:12.9488732Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:12.9489130Z 2025-10-10T01:51:12.9489496Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9489862Z 2025-10-10T01:51:12.9490482Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:12.9491040Z 2025-10-10T01:51:12.9491451Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:12.9491841Z 2025-10-10T01:51:12.9492422Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:12.9492998Z 2025-10-10T01:51:12.9493358Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:12.9493764Z 2025-10-10T01:51:12.9494337Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9494652Z 2025-10-10T01:51:12.9495180Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:12.9495648Z 2025-10-10T01:51:12.9495904Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] self._init_executor() 2025-10-10T01:51:12.9496382Z 2025-10-10T01:51:12.9496936Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:12.9497729Z 2025-10-10T01:51:12.9498141Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:12.9498751Z 2025-10-10T01:51:12.9499980Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:12.9500746Z 2025-10-10T01:51:12.9501144Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:12.9501497Z 2025-10-10T01:51:12.9501790Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9502095Z 2025-10-10T01:51:12.9502596Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:12.9503047Z 2025-10-10T01:51:12.9503317Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:12.9503592Z 2025-10-10T01:51:12.9503964Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9504236Z 2025-10-10T01:51:12.9504773Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:12.9505249Z 2025-10-10T01:51:12.9505545Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:12.9505851Z 2025-10-10T01:51:12.9506133Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9506411Z 2025-10-10T01:51:12.9506951Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:12.9507443Z 2025-10-10T01:51:12.9507763Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:12.9508070Z 2025-10-10T01:51:12.9508357Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9508632Z 2025-10-10T01:51:12.9509079Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:12.9509489Z 2025-10-10T01:51:12.9509838Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:12.9510174Z 2025-10-10T01:51:12.9510472Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9510745Z 2025-10-10T01:51:12.9511160Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:12.9511522Z 2025-10-10T01:51:12.9511887Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:12.9512227Z 2025-10-10T01:51:12.9512597Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:12.9512958Z 2025-10-10T01:51:12.9513298Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:12.9513691Z 2025-10-10T01:51:12.9514061Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:12.9514410Z 2025-10-10T01:51:12.9514844Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:12.9515201Z 2025-10-10T01:51:12.9515702Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:12.9516161Z 2025-10-10T01:51:12.9516516Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:12.9516856Z 2025-10-10T01:51:12.9517378Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:12.9517858Z 2025-10-10T01:51:12.9518230Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:12.9518595Z 2025-10-10T01:51:12.9519325Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:12.9519851Z 2025-10-10T01:51:12.9520161Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:12.9520462Z 2025-10-10T01:51:12.9521086Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:12.9521653Z 2025-10-10T01:51:12.9522021Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:12.9522365Z 2025-10-10T01:51:12.9522631Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9522887Z 2025-10-10T01:51:12.9523449Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:12.9523957Z 2025-10-10T01:51:12.9524257Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:12.9524556Z 2025-10-10T01:51:12.9524831Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9525111Z 2025-10-10T01:51:12.9525712Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:12.9526281Z 2025-10-10T01:51:12.9526614Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:12.9526906Z 2025-10-10T01:51:12.9527166Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9527428Z 2025-10-10T01:51:12.9528013Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:12.9528533Z 2025-10-10T01:51:12.9528915Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:12.9529223Z 2025-10-10T01:51:12.9529498Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9529768Z 2025-10-10T01:51:12.9530351Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:12.9530836Z 2025-10-10T01:51:12.9531115Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:12.9531411Z 2025-10-10T01:51:12.9531669Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9531938Z 2025-10-10T01:51:12.9532462Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:12.9532946Z 2025-10-10T01:51:12.9533250Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:12.9533549Z 2025-10-10T01:51:12.9533783Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:12.9534021Z 2025-10-10T01:51:12.9534552Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:12.9534998Z 2025-10-10T01:51:12.9535247Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] raise RuntimeError( 2025-10-10T01:51:12.9535509Z 2025-10-10T01:51:12.9536075Z (EngineCore_DP0 pid=7569) ERROR 10-10 01:51:12 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:12.9536768Z (EngineCore_DP0 pid=7569) Process EngineCore_DP0: 2025-10-10T01:51:12.9537178Z (EngineCore_DP0 pid=7569) Traceback (most recent call last): 2025-10-10T01:51:12.9537783Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:12.9538300Z (EngineCore_DP0 pid=7569) self.run() 2025-10-10T01:51:12.9538821Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:12.9539379Z (EngineCore_DP0 pid=7569) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:12.9540036Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:12.9540584Z (EngineCore_DP0 pid=7569) raise e 2025-10-10T01:51:12.9541169Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:12.9541799Z (EngineCore_DP0 pid=7569) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:12.9542303Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9542925Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:12.9543577Z (EngineCore_DP0 pid=7569) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:12.9544233Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:12.9544919Z (EngineCore_DP0 pid=7569) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:12.9545382Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9546030Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:12.9546689Z (EngineCore_DP0 pid=7569) self._init_executor() 2025-10-10T01:51:12.9547366Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:12.9548069Z (EngineCore_DP0 pid=7569) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:12.9548801Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:12.9549519Z (EngineCore_DP0 pid=7569) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:12.9550004Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9550645Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:12.9551231Z (EngineCore_DP0 pid=7569) return func(*args, **kwargs) 2025-10-10T01:51:12.9551664Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9552326Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:12.9552945Z (EngineCore_DP0 pid=7569) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:12.9553364Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9554027Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:12.9554677Z (EngineCore_DP0 pid=7569) module = importlib.import_module(module_name) 2025-10-10T01:51:12.9555121Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9555699Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:12.9556316Z (EngineCore_DP0 pid=7569) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:12.9556802Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9557302Z (EngineCore_DP0 pid=7569) File "", line 1387, in _gcd_import 2025-10-10T01:51:12.9557860Z (EngineCore_DP0 pid=7569) File "", line 1360, in _find_and_load 2025-10-10T01:51:12.9558441Z (EngineCore_DP0 pid=7569) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:12.9559063Z (EngineCore_DP0 pid=7569) File "", line 935, in _load_unlocked 2025-10-10T01:51:12.9559689Z (EngineCore_DP0 pid=7569) File "", line 999, in exec_module 2025-10-10T01:51:12.9560309Z (EngineCore_DP0 pid=7569) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:12.9561043Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:12.9561709Z (EngineCore_DP0 pid=7569) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:12.9562512Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:12.9563233Z (EngineCore_DP0 pid=7569) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:12.9564012Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:12.9564761Z (EngineCore_DP0 pid=7569) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:12.9565548Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:12.9566362Z (EngineCore_DP0 pid=7569) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:12.9566854Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9567545Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:12.9568242Z (EngineCore_DP0 pid=7569) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:12.9568682Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9569486Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:12.9570205Z (EngineCore_DP0 pid=7569) return _is_fa2_supported(device)[0] 2025-10-10T01:51:12.9570617Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9571337Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:12.9572053Z (EngineCore_DP0 pid=7569) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:12.9572499Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9573171Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:12.9573810Z (EngineCore_DP0 pid=7569) prop = get_device_properties(device) 2025-10-10T01:51:12.9574225Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:12.9574881Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:12.9575542Z (EngineCore_DP0 pid=7569) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:12.9575931Z (EngineCore_DP0 pid=7569) ^^^^^^^^^^^^ 2025-10-10T01:51:12.9576509Z (EngineCore_DP0 pid=7569) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:12.9577067Z (EngineCore_DP0 pid=7569) raise RuntimeError( 2025-10-10T01:51:12.9577805Z (EngineCore_DP0 pid=7569) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:13.3802789Z FAILED 2025-10-10T01:51:13.3935333Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5_VLForConditionalGeneration] Fork a new process to run a test 7573 2025-10-10T01:51:13.3946172Z Fork a new process to run a test 0 2025-10-10T01:51:13.4231797Z INFO 10-10 01:51:13 [utils.py:233] non-default args: {'load_format': 'dummy', 'max_model_len': 4096, 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2_5_VLForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen2.5-VL-3B-Instruct'} 2025-10-10T01:51:13.5104270Z 2025-10-10T01:51:13.5106590Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:51:13.5107118Z config.json: 1.37kB [00:00, 6.94MB/s] 2025-10-10T01:51:13.6448048Z 2025-10-10T01:51:13.6449539Z preprocessor_config.json: 0% 0.00/350 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:15.1029156Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:15.1029566Z 2025-10-10T01:51:15.1029967Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:15.1030566Z 2025-10-10T01:51:15.1031266Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:15.1031902Z 2025-10-10T01:51:15.1032295Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:15.1032672Z 2025-10-10T01:51:15.1033020Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1033364Z 2025-10-10T01:51:15.1033974Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:15.1034486Z 2025-10-10T01:51:15.1034864Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:15.1035335Z 2025-10-10T01:51:15.1035938Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:15.1036407Z 2025-10-10T01:51:15.1036736Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:15.1037048Z 2025-10-10T01:51:15.1037447Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1037887Z 2025-10-10T01:51:15.1038755Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:15.1039693Z 2025-10-10T01:51:15.1040180Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] self._init_executor() 2025-10-10T01:51:15.1040609Z 2025-10-10T01:51:15.1041375Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:15.1041892Z 2025-10-10T01:51:15.1042232Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:15.1042547Z 2025-10-10T01:51:15.1043092Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:15.1043591Z 2025-10-10T01:51:15.1043986Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:15.1044384Z 2025-10-10T01:51:15.1044852Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1045210Z 2025-10-10T01:51:15.1045806Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:15.1046317Z 2025-10-10T01:51:15.1046592Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:15.1046873Z 2025-10-10T01:51:15.1047222Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1047491Z 2025-10-10T01:51:15.1048160Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:15.1048710Z 2025-10-10T01:51:15.1049030Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:15.1049319Z 2025-10-10T01:51:15.1049590Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1049874Z 2025-10-10T01:51:15.1050407Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:15.1050904Z 2025-10-10T01:51:15.1051210Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:15.1051517Z 2025-10-10T01:51:15.1051791Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1052070Z 2025-10-10T01:51:15.1052570Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:15.1053020Z 2025-10-10T01:51:15.1053399Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:15.1053797Z 2025-10-10T01:51:15.1054142Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1054479Z 2025-10-10T01:51:15.1054879Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:15.1055277Z 2025-10-10T01:51:15.1055638Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:15.1055985Z 2025-10-10T01:51:15.1056374Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:15.1056733Z 2025-10-10T01:51:15.1057081Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:15.1057418Z 2025-10-10T01:51:15.1057790Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:15.1058144Z 2025-10-10T01:51:15.1058521Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:15.1058882Z 2025-10-10T01:51:15.1059386Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:15.1059864Z 2025-10-10T01:51:15.1060272Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:15.1060620Z 2025-10-10T01:51:15.1061147Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:15.1061634Z 2025-10-10T01:51:15.1062007Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:15.1062418Z 2025-10-10T01:51:15.1062984Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:15.1063500Z 2025-10-10T01:51:15.1063797Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:15.1064134Z 2025-10-10T01:51:15.1064778Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:15.1065354Z 2025-10-10T01:51:15.1065724Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:15.1066091Z 2025-10-10T01:51:15.1066355Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1066619Z 2025-10-10T01:51:15.1067180Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:15.1067709Z 2025-10-10T01:51:15.1068010Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:15.1068364Z 2025-10-10T01:51:15.1068659Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1068939Z 2025-10-10T01:51:15.1069558Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:15.1070112Z 2025-10-10T01:51:15.1070392Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:15.1070675Z 2025-10-10T01:51:15.1070938Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1071203Z 2025-10-10T01:51:15.1071795Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:15.1072330Z 2025-10-10T01:51:15.1072655Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:15.1072964Z 2025-10-10T01:51:15.1073237Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1073515Z 2025-10-10T01:51:15.1074041Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:15.1074525Z 2025-10-10T01:51:15.1074804Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:15.1075091Z 2025-10-10T01:51:15.1075422Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1075697Z 2025-10-10T01:51:15.1076227Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:15.1076704Z 2025-10-10T01:51:15.1077017Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:15.1077317Z 2025-10-10T01:51:15.1077548Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:15.1077833Z 2025-10-10T01:51:15.1078317Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:15.1078763Z 2025-10-10T01:51:15.1079189Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] raise RuntimeError( 2025-10-10T01:51:15.1079455Z 2025-10-10T01:51:15.1080030Z (EngineCore_DP0 pid=7581) ERROR 10-10 01:51:15 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:15.1080731Z (EngineCore_DP0 pid=7581) Process EngineCore_DP0: 2025-10-10T01:51:15.1081128Z (EngineCore_DP0 pid=7581) Traceback (most recent call last): 2025-10-10T01:51:15.1081758Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:15.1082284Z (EngineCore_DP0 pid=7581) self.run() 2025-10-10T01:51:15.1082807Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:15.1083372Z (EngineCore_DP0 pid=7581) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:15.1084114Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:15.1084673Z (EngineCore_DP0 pid=7581) raise e 2025-10-10T01:51:15.1085260Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:15.1085889Z (EngineCore_DP0 pid=7581) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:15.1086349Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1086970Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:15.1087609Z (EngineCore_DP0 pid=7581) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:15.1088276Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:15.1088891Z (EngineCore_DP0 pid=7581) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:15.1089341Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1089991Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:15.1090579Z (EngineCore_DP0 pid=7581) self._init_executor() 2025-10-10T01:51:15.1091238Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:15.1091918Z (EngineCore_DP0 pid=7581) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:15.1092698Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:15.1093440Z (EngineCore_DP0 pid=7581) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:15.1093954Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1094604Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:15.1095253Z (EngineCore_DP0 pid=7581) return func(*args, **kwargs) 2025-10-10T01:51:15.1095641Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1096661Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:15.1097447Z (EngineCore_DP0 pid=7581) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:15.1097881Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1098545Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:15.1099208Z (EngineCore_DP0 pid=7581) module = importlib.import_module(module_name) 2025-10-10T01:51:15.1099658Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1100242Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:15.1100869Z (EngineCore_DP0 pid=7581) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:15.1101368Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1101940Z (EngineCore_DP0 pid=7581) File "", line 1387, in _gcd_import 2025-10-10T01:51:15.1102513Z (EngineCore_DP0 pid=7581) File "", line 1360, in _find_and_load 2025-10-10T01:51:15.1103116Z (EngineCore_DP0 pid=7581) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:15.1103705Z (EngineCore_DP0 pid=7581) File "", line 935, in _load_unlocked 2025-10-10T01:51:15.1104278Z (EngineCore_DP0 pid=7581) File "", line 999, in exec_module 2025-10-10T01:51:15.1104873Z (EngineCore_DP0 pid=7581) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:15.1105617Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:15.1106311Z (EngineCore_DP0 pid=7581) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:15.1107047Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:15.1107765Z (EngineCore_DP0 pid=7581) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:15.1108561Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:15.1109232Z (EngineCore_DP0 pid=7581) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:15.1110017Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:15.1110901Z (EngineCore_DP0 pid=7581) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:15.1111390Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1112080Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:15.1112779Z (EngineCore_DP0 pid=7581) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:15.1113224Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1114074Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:15.1114771Z (EngineCore_DP0 pid=7581) return _is_fa2_supported(device)[0] 2025-10-10T01:51:15.1115245Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1115958Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:15.1116670Z (EngineCore_DP0 pid=7581) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:15.1117116Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1117782Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:15.1118412Z (EngineCore_DP0 pid=7581) prop = get_device_properties(device) 2025-10-10T01:51:15.1118837Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:15.1119562Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:15.1120277Z (EngineCore_DP0 pid=7581) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:15.1120682Z (EngineCore_DP0 pid=7581) ^^^^^^^^^^^^ 2025-10-10T01:51:15.1121265Z (EngineCore_DP0 pid=7581) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:15.1121832Z (EngineCore_DP0 pid=7581) raise RuntimeError( 2025-10-10T01:51:15.1122511Z (EngineCore_DP0 pid=7581) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:15.5199876Z FAILED 2025-10-10T01:51:15.5329023Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForCausalLM] Fork a new process to run a test 7585 2025-10-10T01:51:15.5340895Z Fork a new process to run a test 0 2025-10-10T01:51:15.5615506Z INFO 10-10 01:51:15 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen2-0.5B-Instruct'} 2025-10-10T01:51:15.6798216Z 2025-10-10T01:51:15.6799892Z config.json: 0% 0.00/659 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:23.5005652Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:23.5006304Z 2025-10-10T01:51:23.5007000Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:23.5007645Z 2025-10-10T01:51:23.5008426Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:23.5009055Z 2025-10-10T01:51:23.5009453Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:23.5009839Z 2025-10-10T01:51:23.5010197Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5010822Z 2025-10-10T01:51:23.5011472Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:23.5012010Z 2025-10-10T01:51:23.5012605Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:23.5013163Z 2025-10-10T01:51:23.5013755Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:23.5014324Z 2025-10-10T01:51:23.5014655Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:23.5014957Z 2025-10-10T01:51:23.5015250Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5015620Z 2025-10-10T01:51:23.5016413Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:23.5017238Z 2025-10-10T01:51:23.5017679Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] self._init_executor() 2025-10-10T01:51:23.5018131Z 2025-10-10T01:51:23.5019103Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:23.5019995Z 2025-10-10T01:51:23.5020561Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:23.5021100Z 2025-10-10T01:51:23.5021700Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:23.5022331Z 2025-10-10T01:51:23.5022697Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:23.5023043Z 2025-10-10T01:51:23.5023340Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5023624Z 2025-10-10T01:51:23.5024139Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:23.5024624Z 2025-10-10T01:51:23.5024889Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:23.5025156Z 2025-10-10T01:51:23.5025427Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5025685Z 2025-10-10T01:51:23.5026215Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:23.5026689Z 2025-10-10T01:51:23.5026999Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:23.5027289Z 2025-10-10T01:51:23.5027571Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5027841Z 2025-10-10T01:51:23.5028408Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:23.5028912Z 2025-10-10T01:51:23.5029276Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:23.5029589Z 2025-10-10T01:51:23.5029879Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5030162Z 2025-10-10T01:51:23.5030615Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:23.5031037Z 2025-10-10T01:51:23.5031400Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:23.5031782Z 2025-10-10T01:51:23.5032084Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5032360Z 2025-10-10T01:51:23.5032707Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:23.5033089Z 2025-10-10T01:51:23.5033436Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:23.5033786Z 2025-10-10T01:51:23.5034154Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:23.5034516Z 2025-10-10T01:51:23.5034855Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:23.5035204Z 2025-10-10T01:51:23.5035564Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:23.5035915Z 2025-10-10T01:51:23.5036295Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:23.5036652Z 2025-10-10T01:51:23.5037203Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:23.5037674Z 2025-10-10T01:51:23.5038040Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:23.5038372Z 2025-10-10T01:51:23.5038902Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:23.5039510Z 2025-10-10T01:51:23.5039896Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:23.5040251Z 2025-10-10T01:51:23.5040819Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:23.5041325Z 2025-10-10T01:51:23.5041616Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:23.5041929Z 2025-10-10T01:51:23.5042545Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:23.5043123Z 2025-10-10T01:51:23.5043495Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:23.5043847Z 2025-10-10T01:51:23.5044110Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5044418Z 2025-10-10T01:51:23.5045000Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:23.5045514Z 2025-10-10T01:51:23.5045821Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:23.5046115Z 2025-10-10T01:51:23.5046396Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5046712Z 2025-10-10T01:51:23.5047316Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:23.5047865Z 2025-10-10T01:51:23.5048155Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:23.5048485Z 2025-10-10T01:51:23.5048756Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5049023Z 2025-10-10T01:51:23.5049608Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:23.5050139Z 2025-10-10T01:51:23.5050452Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:23.5050762Z 2025-10-10T01:51:23.5051036Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5051304Z 2025-10-10T01:51:23.5051842Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:23.5052325Z 2025-10-10T01:51:23.5052682Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:23.5052971Z 2025-10-10T01:51:23.5053246Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5053514Z 2025-10-10T01:51:23.5054041Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:23.5054524Z 2025-10-10T01:51:23.5054833Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:23.5055143Z 2025-10-10T01:51:23.5055366Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:23.5055630Z 2025-10-10T01:51:23.5056124Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:23.5056572Z 2025-10-10T01:51:23.5056833Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] raise RuntimeError( 2025-10-10T01:51:23.5057091Z 2025-10-10T01:51:23.5057668Z (EngineCore_DP0 pid=7643) ERROR 10-10 01:51:23 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:23.5058366Z (EngineCore_DP0 pid=7643) Process EngineCore_DP0: 2025-10-10T01:51:23.5058768Z (EngineCore_DP0 pid=7643) Traceback (most recent call last): 2025-10-10T01:51:23.5059429Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:23.5059961Z (EngineCore_DP0 pid=7643) self.run() 2025-10-10T01:51:23.5060501Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:23.5061078Z (EngineCore_DP0 pid=7643) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:23.5061758Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:23.5062330Z (EngineCore_DP0 pid=7643) raise e 2025-10-10T01:51:23.5062985Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:23.5063632Z (EngineCore_DP0 pid=7643) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:23.5064094Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5064773Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:23.5065423Z (EngineCore_DP0 pid=7643) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:23.5066086Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:23.5066712Z (EngineCore_DP0 pid=7643) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:23.5067182Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5067840Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:23.5068422Z (EngineCore_DP0 pid=7643) self._init_executor() 2025-10-10T01:51:23.5069130Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:23.5069835Z (EngineCore_DP0 pid=7643) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:23.5070557Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:23.5071260Z (EngineCore_DP0 pid=7643) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:23.5071777Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5072430Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:23.5073032Z (EngineCore_DP0 pid=7643) return func(*args, **kwargs) 2025-10-10T01:51:23.5073433Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5074073Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:23.5074705Z (EngineCore_DP0 pid=7643) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:23.5075138Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5075820Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:23.5076476Z (EngineCore_DP0 pid=7643) module = importlib.import_module(module_name) 2025-10-10T01:51:23.5076910Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5077539Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:23.5078182Z (EngineCore_DP0 pid=7643) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:23.5078685Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5079238Z (EngineCore_DP0 pid=7643) File "", line 1387, in _gcd_import 2025-10-10T01:51:23.5079803Z (EngineCore_DP0 pid=7643) File "", line 1360, in _find_and_load 2025-10-10T01:51:23.5080447Z (EngineCore_DP0 pid=7643) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:23.5081025Z (EngineCore_DP0 pid=7643) File "", line 935, in _load_unlocked 2025-10-10T01:51:23.5081597Z (EngineCore_DP0 pid=7643) File "", line 999, in exec_module 2025-10-10T01:51:23.5082256Z (EngineCore_DP0 pid=7643) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:23.5083014Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:23.5083686Z (EngineCore_DP0 pid=7643) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:23.5084407Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:23.5085131Z (EngineCore_DP0 pid=7643) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:23.5085916Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:23.5086589Z (EngineCore_DP0 pid=7643) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:23.5087422Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:23.5088259Z (EngineCore_DP0 pid=7643) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:23.5088738Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5089440Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:23.5090126Z (EngineCore_DP0 pid=7643) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:23.5090569Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5091331Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:23.5092043Z (EngineCore_DP0 pid=7643) return _is_fa2_supported(device)[0] 2025-10-10T01:51:23.5092458Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5093180Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:23.5093892Z (EngineCore_DP0 pid=7643) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:23.5094336Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5095056Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:23.5095714Z (EngineCore_DP0 pid=7643) prop = get_device_properties(device) 2025-10-10T01:51:23.5096356Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:23.5097034Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:23.5097708Z (EngineCore_DP0 pid=7643) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:23.5098095Z (EngineCore_DP0 pid=7643) ^^^^^^^^^^^^ 2025-10-10T01:51:23.5098791Z (EngineCore_DP0 pid=7643) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:23.5099352Z (EngineCore_DP0 pid=7643) raise RuntimeError( 2025-10-10T01:51:23.5100056Z (EngineCore_DP0 pid=7643) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:23.9092534Z FAILED 2025-10-10T01:51:23.9223279Z models/test_initialization.py::test_can_initialize_large_subset[Idefics3ForConditionalGeneration] Fork a new process to run a test 7647 2025-10-10T01:51:23.9234393Z Fork a new process to run a test 0 2025-10-10T01:51:23.9507942Z INFO 10-10 01:51:23 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Idefics3ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'HuggingFaceM4/Idefics3-8B-Llama3'} 2025-10-10T01:51:24.0512006Z 2025-10-10T01:51:24.0514573Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:51:24.0515037Z config.json: 7.18kB [00:00, 31.3MB/s] 2025-10-10T01:51:24.1931016Z 2025-10-10T01:51:24.1932748Z preprocessor_config.json: 0% 0.00/435 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:32.3783709Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:32.3784363Z 2025-10-10T01:51:32.3784919Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:32.3785307Z 2025-10-10T01:51:32.3786027Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:32.3786859Z 2025-10-10T01:51:32.3787295Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:32.3787682Z 2025-10-10T01:51:32.3788025Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3788379Z 2025-10-10T01:51:32.3788991Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:32.3789754Z 2025-10-10T01:51:32.3790175Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:32.3790513Z 2025-10-10T01:51:32.3791008Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:32.3791460Z 2025-10-10T01:51:32.3791784Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:32.3792090Z 2025-10-10T01:51:32.3792369Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3792801Z 2025-10-10T01:51:32.3793795Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:32.3794599Z 2025-10-10T01:51:32.3795098Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] self._init_executor() 2025-10-10T01:51:32.3795575Z 2025-10-10T01:51:32.3796585Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:32.3797119Z 2025-10-10T01:51:32.3797472Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:32.3797800Z 2025-10-10T01:51:32.3798342Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:32.3798941Z 2025-10-10T01:51:32.3799420Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:32.3799766Z 2025-10-10T01:51:32.3800070Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3800447Z 2025-10-10T01:51:32.3800947Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:32.3801434Z 2025-10-10T01:51:32.3801701Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:32.3801999Z 2025-10-10T01:51:32.3802266Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3802521Z 2025-10-10T01:51:32.3803041Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:32.3803526Z 2025-10-10T01:51:32.3803824Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:32.3804128Z 2025-10-10T01:51:32.3804456Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3804735Z 2025-10-10T01:51:32.3805265Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:32.3805752Z 2025-10-10T01:51:32.3806064Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:32.3806370Z 2025-10-10T01:51:32.3806644Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3806923Z 2025-10-10T01:51:32.3807355Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:32.3807777Z 2025-10-10T01:51:32.3808127Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:32.3808454Z 2025-10-10T01:51:32.3808744Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3809016Z 2025-10-10T01:51:32.3809358Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:32.3809682Z 2025-10-10T01:51:32.3810027Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:32.3810376Z 2025-10-10T01:51:32.3810762Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:32.3811130Z 2025-10-10T01:51:32.3811532Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:32.3811877Z 2025-10-10T01:51:32.3812237Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:32.3812593Z 2025-10-10T01:51:32.3812960Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:32.3813317Z 2025-10-10T01:51:32.3813929Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:32.3814389Z 2025-10-10T01:51:32.3814741Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:32.3815126Z 2025-10-10T01:51:32.3815651Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:32.3816124Z 2025-10-10T01:51:32.3816497Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:32.3816849Z 2025-10-10T01:51:32.3817390Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:32.3817894Z 2025-10-10T01:51:32.3818191Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:32.3818490Z 2025-10-10T01:51:32.3819115Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:32.3819740Z 2025-10-10T01:51:32.3820104Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:32.3820462Z 2025-10-10T01:51:32.3820724Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3820985Z 2025-10-10T01:51:32.3821552Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:32.3822070Z 2025-10-10T01:51:32.3822367Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:32.3822656Z 2025-10-10T01:51:32.3822947Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3823217Z 2025-10-10T01:51:32.3823808Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:32.3824355Z 2025-10-10T01:51:32.3824633Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:32.3824913Z 2025-10-10T01:51:32.3825168Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3825433Z 2025-10-10T01:51:32.3826002Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:32.3826536Z 2025-10-10T01:51:32.3826904Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:32.3827219Z 2025-10-10T01:51:32.3827498Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3827769Z 2025-10-10T01:51:32.3828292Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:32.3828809Z 2025-10-10T01:51:32.3829090Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:32.3829372Z 2025-10-10T01:51:32.3829631Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3829898Z 2025-10-10T01:51:32.3830471Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:32.3830950Z 2025-10-10T01:51:32.3831255Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:32.3831559Z 2025-10-10T01:51:32.3831779Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:32.3832023Z 2025-10-10T01:51:32.3832500Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:32.3832955Z 2025-10-10T01:51:32.3833205Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] raise RuntimeError( 2025-10-10T01:51:32.3833456Z 2025-10-10T01:51:32.3834063Z (EngineCore_DP0 pid=7705) ERROR 10-10 01:51:32 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:32.3834773Z (EngineCore_DP0 pid=7705) Process EngineCore_DP0: 2025-10-10T01:51:32.3835166Z (EngineCore_DP0 pid=7705) Traceback (most recent call last): 2025-10-10T01:51:32.3835765Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:32.3836275Z (EngineCore_DP0 pid=7705) self.run() 2025-10-10T01:51:32.3836833Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:32.3837398Z (EngineCore_DP0 pid=7705) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:32.3838072Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:32.3838631Z (EngineCore_DP0 pid=7705) raise e 2025-10-10T01:51:32.3839281Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:32.3839921Z (EngineCore_DP0 pid=7705) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:32.3840361Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3840971Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:32.3841611Z (EngineCore_DP0 pid=7705) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:32.3842276Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:32.3842946Z (EngineCore_DP0 pid=7705) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:32.3843414Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3844074Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:32.3844659Z (EngineCore_DP0 pid=7705) self._init_executor() 2025-10-10T01:51:32.3845329Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:32.3846081Z (EngineCore_DP0 pid=7705) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:32.3846803Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:32.3847558Z (EngineCore_DP0 pid=7705) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:32.3848046Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3848675Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:32.3849259Z (EngineCore_DP0 pid=7705) return func(*args, **kwargs) 2025-10-10T01:51:32.3849639Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3850265Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:32.3850889Z (EngineCore_DP0 pid=7705) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:32.3851307Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3852024Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:32.3852677Z (EngineCore_DP0 pid=7705) module = importlib.import_module(module_name) 2025-10-10T01:51:32.3853113Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3853705Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:32.3854325Z (EngineCore_DP0 pid=7705) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:32.3854811Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3855302Z (EngineCore_DP0 pid=7705) File "", line 1387, in _gcd_import 2025-10-10T01:51:32.3855879Z (EngineCore_DP0 pid=7705) File "", line 1360, in _find_and_load 2025-10-10T01:51:32.3856456Z (EngineCore_DP0 pid=7705) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:32.3857028Z (EngineCore_DP0 pid=7705) File "", line 935, in _load_unlocked 2025-10-10T01:51:32.3857595Z (EngineCore_DP0 pid=7705) File "", line 999, in exec_module 2025-10-10T01:51:32.3858189Z (EngineCore_DP0 pid=7705) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:32.3858922Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:32.3859604Z (EngineCore_DP0 pid=7705) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:32.3860381Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:32.3861121Z (EngineCore_DP0 pid=7705) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:32.3861921Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:32.3862592Z (EngineCore_DP0 pid=7705) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:32.3863416Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:32.3864222Z (EngineCore_DP0 pid=7705) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:32.3864758Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3865446Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:32.3866130Z (EngineCore_DP0 pid=7705) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:32.3866577Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3867310Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:32.3868006Z (EngineCore_DP0 pid=7705) return _is_fa2_supported(device)[0] 2025-10-10T01:51:32.3868414Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3869129Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:32.3870014Z (EngineCore_DP0 pid=7705) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:32.3870550Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3871342Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:32.3872039Z (EngineCore_DP0 pid=7705) prop = get_device_properties(device) 2025-10-10T01:51:32.3872452Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:32.3873105Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:32.3873773Z (EngineCore_DP0 pid=7705) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:32.3874167Z (EngineCore_DP0 pid=7705) ^^^^^^^^^^^^ 2025-10-10T01:51:32.3874744Z (EngineCore_DP0 pid=7705) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:32.3875314Z (EngineCore_DP0 pid=7705) raise RuntimeError( 2025-10-10T01:51:32.3875998Z (EngineCore_DP0 pid=7705) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:32.8018916Z FAILED 2025-10-10T01:51:33.0422758Z models/test_initialization.py::test_can_initialize_large_subset[Lfm2ForCausalLM] SKIPPED 2025-10-10T01:51:33.0552227Z models/test_initialization.py::test_can_initialize_large_subset[Qwen3ForCausalLM] Fork a new process to run a test 7709 2025-10-10T01:51:33.0563549Z Fork a new process to run a test 0 2025-10-10T01:51:33.0842893Z INFO 10-10 01:51:33 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen3-8B'} 2025-10-10T01:51:33.1643810Z 2025-10-10T01:51:33.1645493Z config.json: 0% 0.00/728 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:41.2489608Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:41.2490063Z 2025-10-10T01:51:41.2490468Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:41.2490836Z 2025-10-10T01:51:41.2491518Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:41.2492374Z 2025-10-10T01:51:41.2492778Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:41.2493155Z 2025-10-10T01:51:41.2493525Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2494017Z 2025-10-10T01:51:41.2494654Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:41.2495250Z 2025-10-10T01:51:41.2495873Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:41.2496466Z 2025-10-10T01:51:41.2497066Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:41.2497671Z 2025-10-10T01:51:41.2498059Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:41.2498364Z 2025-10-10T01:51:41.2498646Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2498931Z 2025-10-10T01:51:41.2499822Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:41.2500320Z 2025-10-10T01:51:41.2500606Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] self._init_executor() 2025-10-10T01:51:41.2501034Z 2025-10-10T01:51:41.2501679Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:41.2502482Z 2025-10-10T01:51:41.2502831Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:41.2503178Z 2025-10-10T01:51:41.2503841Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:41.2504355Z 2025-10-10T01:51:41.2504715Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:41.2505049Z 2025-10-10T01:51:41.2505343Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2505621Z 2025-10-10T01:51:41.2506110Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:41.2506570Z 2025-10-10T01:51:41.2506834Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:41.2507108Z 2025-10-10T01:51:41.2507464Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2507743Z 2025-10-10T01:51:41.2508271Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:41.2508749Z 2025-10-10T01:51:41.2509045Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:41.2509342Z 2025-10-10T01:51:41.2509619Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2509994Z 2025-10-10T01:51:41.2510532Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:41.2511039Z 2025-10-10T01:51:41.2511352Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:41.2511736Z 2025-10-10T01:51:41.2512022Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2512307Z 2025-10-10T01:51:41.2512744Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:41.2513158Z 2025-10-10T01:51:41.2513498Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:41.2513833Z 2025-10-10T01:51:41.2514115Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2514398Z 2025-10-10T01:51:41.2514741Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:41.2515073Z 2025-10-10T01:51:41.2515504Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:41.2515846Z 2025-10-10T01:51:41.2516237Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:41.2516596Z 2025-10-10T01:51:41.2516944Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:41.2517278Z 2025-10-10T01:51:41.2517635Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:41.2517998Z 2025-10-10T01:51:41.2518372Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:41.2518739Z 2025-10-10T01:51:41.2519391Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:41.2519859Z 2025-10-10T01:51:41.2520216Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:41.2520556Z 2025-10-10T01:51:41.2521073Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:41.2521562Z 2025-10-10T01:51:41.2521937Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:41.2522298Z 2025-10-10T01:51:41.2522932Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:41.2523442Z 2025-10-10T01:51:41.2523743Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:41.2524038Z 2025-10-10T01:51:41.2524662Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:41.2525289Z 2025-10-10T01:51:41.2525656Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:41.2526015Z 2025-10-10T01:51:41.2526270Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2526577Z 2025-10-10T01:51:41.2527141Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:41.2527671Z 2025-10-10T01:51:41.2527967Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:41.2528258Z 2025-10-10T01:51:41.2528529Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2528801Z 2025-10-10T01:51:41.2529403Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:41.2529951Z 2025-10-10T01:51:41.2530235Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:41.2530518Z 2025-10-10T01:51:41.2530831Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2531099Z 2025-10-10T01:51:41.2531675Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:41.2532209Z 2025-10-10T01:51:41.2532524Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:41.2532855Z 2025-10-10T01:51:41.2533123Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2533402Z 2025-10-10T01:51:41.2533924Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:41.2534414Z 2025-10-10T01:51:41.2534691Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:41.2534973Z 2025-10-10T01:51:41.2535235Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2535500Z 2025-10-10T01:51:41.2536024Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:41.2536504Z 2025-10-10T01:51:41.2536813Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:41.2537113Z 2025-10-10T01:51:41.2537384Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:41.2537638Z 2025-10-10T01:51:41.2538140Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:41.2538600Z 2025-10-10T01:51:41.2538846Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] raise RuntimeError( 2025-10-10T01:51:41.2539152Z 2025-10-10T01:51:41.2539714Z (EngineCore_DP0 pid=7787) ERROR 10-10 01:51:41 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:41.2540459Z (EngineCore_DP0 pid=7787) Process EngineCore_DP0: 2025-10-10T01:51:41.2540851Z (EngineCore_DP0 pid=7787) Traceback (most recent call last): 2025-10-10T01:51:41.2541481Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:41.2542044Z (EngineCore_DP0 pid=7787) self.run() 2025-10-10T01:51:41.2542573Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:41.2543146Z (EngineCore_DP0 pid=7787) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:41.2543811Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:41.2544358Z (EngineCore_DP0 pid=7787) raise e 2025-10-10T01:51:41.2544947Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:41.2545569Z (EngineCore_DP0 pid=7787) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:41.2546012Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2546678Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:41.2547324Z (EngineCore_DP0 pid=7787) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:41.2547999Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:41.2548609Z (EngineCore_DP0 pid=7787) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:41.2549068Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2549724Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:41.2550309Z (EngineCore_DP0 pid=7787) self._init_executor() 2025-10-10T01:51:41.2550977Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:41.2551655Z (EngineCore_DP0 pid=7787) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:41.2552376Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:41.2553089Z (EngineCore_DP0 pid=7787) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:41.2553581Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2554229Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:41.2554859Z (EngineCore_DP0 pid=7787) return func(*args, **kwargs) 2025-10-10T01:51:41.2555247Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2555890Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:41.2556521Z (EngineCore_DP0 pid=7787) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:41.2556939Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2557598Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:41.2558298Z (EngineCore_DP0 pid=7787) module = importlib.import_module(module_name) 2025-10-10T01:51:41.2558737Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2559439Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:41.2560064Z (EngineCore_DP0 pid=7787) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:41.2560552Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2561042Z (EngineCore_DP0 pid=7787) File "", line 1387, in _gcd_import 2025-10-10T01:51:41.2561589Z (EngineCore_DP0 pid=7787) File "", line 1360, in _find_and_load 2025-10-10T01:51:41.2562174Z (EngineCore_DP0 pid=7787) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:41.2562764Z (EngineCore_DP0 pid=7787) File "", line 935, in _load_unlocked 2025-10-10T01:51:41.2563335Z (EngineCore_DP0 pid=7787) File "", line 999, in exec_module 2025-10-10T01:51:41.2563973Z (EngineCore_DP0 pid=7787) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:41.2564724Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:41.2565399Z (EngineCore_DP0 pid=7787) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:41.2566111Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:41.2566830Z (EngineCore_DP0 pid=7787) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:41.2567601Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:41.2568281Z (EngineCore_DP0 pid=7787) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:41.2569076Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:41.2569882Z (EngineCore_DP0 pid=7787) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:41.2570361Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2571058Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:41.2571737Z (EngineCore_DP0 pid=7787) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:41.2572181Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2572978Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:41.2573687Z (EngineCore_DP0 pid=7787) return _is_fa2_supported(device)[0] 2025-10-10T01:51:41.2574101Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2574803Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:41.2575566Z (EngineCore_DP0 pid=7787) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:41.2576029Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2576701Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:41.2577387Z (EngineCore_DP0 pid=7787) prop = get_device_properties(device) 2025-10-10T01:51:41.2577805Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:41.2578464Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:41.2579125Z (EngineCore_DP0 pid=7787) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:41.2579520Z (EngineCore_DP0 pid=7787) ^^^^^^^^^^^^ 2025-10-10T01:51:41.2580108Z (EngineCore_DP0 pid=7787) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:41.2580662Z (EngineCore_DP0 pid=7787) raise RuntimeError( 2025-10-10T01:51:41.2581347Z (EngineCore_DP0 pid=7787) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:41.6652342Z FAILED 2025-10-10T01:51:41.6782838Z models/test_initialization.py::test_can_initialize_large_subset[Dots1ForCausalLM] Fork a new process to run a test 7791 2025-10-10T01:51:41.6794745Z Fork a new process to run a test 0 2025-10-10T01:51:41.7073653Z INFO 10-10 01:51:41 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Dots1ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'rednote-hilab/dots.llm1.inst'} 2025-10-10T01:51:41.8969297Z 2025-10-10T01:51:41.8970037Z config.json: 0% 0.00/988 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:49.6751324Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:49.6751738Z 2025-10-10T01:51:49.6752115Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:49.6752468Z 2025-10-10T01:51:49.6753132Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:49.6753744Z 2025-10-10T01:51:49.6754140Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:49.6754521Z 2025-10-10T01:51:49.6754865Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6755207Z 2025-10-10T01:51:49.6755761Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:49.6756211Z 2025-10-10T01:51:49.6756537Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:49.6756852Z 2025-10-10T01:51:49.6757561Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:49.6758025Z 2025-10-10T01:51:49.6758341Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:49.6758640Z 2025-10-10T01:51:49.6758918Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6759327Z 2025-10-10T01:51:49.6759836Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:49.6760393Z 2025-10-10T01:51:49.6760647Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] self._init_executor() 2025-10-10T01:51:49.6760899Z 2025-10-10T01:51:49.6761589Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:49.6762259Z 2025-10-10T01:51:49.6762730Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:49.6763058Z 2025-10-10T01:51:49.6763867Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:49.6764507Z 2025-10-10T01:51:49.6764947Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:49.6765365Z 2025-10-10T01:51:49.6765741Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6766034Z 2025-10-10T01:51:49.6766658Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:49.6767137Z 2025-10-10T01:51:49.6767536Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:49.6767818Z 2025-10-10T01:51:49.6768089Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6768343Z 2025-10-10T01:51:49.6768864Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:49.6769337Z 2025-10-10T01:51:49.6769623Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:49.6769915Z 2025-10-10T01:51:49.6770176Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6770457Z 2025-10-10T01:51:49.6770989Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:49.6771493Z 2025-10-10T01:51:49.6771800Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:49.6772098Z 2025-10-10T01:51:49.6772373Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6772655Z 2025-10-10T01:51:49.6773098Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:49.6773507Z 2025-10-10T01:51:49.6773951Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:49.6774293Z 2025-10-10T01:51:49.6774582Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6774865Z 2025-10-10T01:51:49.6775201Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:49.6775534Z 2025-10-10T01:51:49.6775895Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:49.6776274Z 2025-10-10T01:51:49.6776652Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:49.6777006Z 2025-10-10T01:51:49.6777356Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:49.6777751Z 2025-10-10T01:51:49.6778119Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:49.6778481Z 2025-10-10T01:51:49.6778852Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:49.6779216Z 2025-10-10T01:51:49.6779716Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:49.6780183Z 2025-10-10T01:51:49.6780532Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:49.6780875Z 2025-10-10T01:51:49.6781666Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:49.6782251Z 2025-10-10T01:51:49.6782662Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:49.6783027Z 2025-10-10T01:51:49.6783593Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:49.6784101Z 2025-10-10T01:51:49.6784412Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:49.6784708Z 2025-10-10T01:51:49.6785442Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:49.6794636Z 2025-10-10T01:51:49.6795178Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:49.6795639Z 2025-10-10T01:51:49.6795971Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6796753Z 2025-10-10T01:51:49.6797408Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:49.6797953Z 2025-10-10T01:51:49.6798259Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:49.6798564Z 2025-10-10T01:51:49.6798851Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6799272Z 2025-10-10T01:51:49.6800038Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:49.6800614Z 2025-10-10T01:51:49.6800909Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:49.6801203Z 2025-10-10T01:51:49.6801471Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6801819Z 2025-10-10T01:51:49.6802405Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:49.6802932Z 2025-10-10T01:51:49.6803262Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:49.6803650Z 2025-10-10T01:51:49.6803937Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6804210Z 2025-10-10T01:51:49.6804735Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:49.6805225Z 2025-10-10T01:51:49.6805510Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:49.6805804Z 2025-10-10T01:51:49.6806064Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6806346Z 2025-10-10T01:51:49.6806880Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:49.6807373Z 2025-10-10T01:51:49.6807749Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:49.6808070Z 2025-10-10T01:51:49.6808307Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:49.6808549Z 2025-10-10T01:51:49.6809056Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:49.6809533Z 2025-10-10T01:51:49.6809789Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] raise RuntimeError( 2025-10-10T01:51:49.6810049Z 2025-10-10T01:51:49.6810623Z (EngineCore_DP0 pid=7849) ERROR 10-10 01:51:49 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:49.6811335Z (EngineCore_DP0 pid=7849) Process EngineCore_DP0: 2025-10-10T01:51:49.6811741Z (EngineCore_DP0 pid=7849) Traceback (most recent call last): 2025-10-10T01:51:49.6812355Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:49.6812873Z (EngineCore_DP0 pid=7849) self.run() 2025-10-10T01:51:49.6813395Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:49.6813964Z (EngineCore_DP0 pid=7849) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:49.6814648Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:49.6815199Z (EngineCore_DP0 pid=7849) raise e 2025-10-10T01:51:49.6815859Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:49.6816507Z (EngineCore_DP0 pid=7849) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:49.6816953Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6817583Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:49.6818225Z (EngineCore_DP0 pid=7849) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:49.6818934Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:49.6819553Z (EngineCore_DP0 pid=7849) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:49.6820056Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6820714Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:49.6821312Z (EngineCore_DP0 pid=7849) self._init_executor() 2025-10-10T01:51:49.6821981Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:49.6822669Z (EngineCore_DP0 pid=7849) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:49.6823392Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:49.6824107Z (EngineCore_DP0 pid=7849) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:49.6824612Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6825303Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:49.6825898Z (EngineCore_DP0 pid=7849) return func(*args, **kwargs) 2025-10-10T01:51:49.6826275Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6826915Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:49.6827550Z (EngineCore_DP0 pid=7849) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:49.6827983Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6828663Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:49.6829318Z (EngineCore_DP0 pid=7849) module = importlib.import_module(module_name) 2025-10-10T01:51:49.6829764Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6830353Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:49.6830999Z (EngineCore_DP0 pid=7849) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:49.6831496Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6831987Z (EngineCore_DP0 pid=7849) File "", line 1387, in _gcd_import 2025-10-10T01:51:49.6832548Z (EngineCore_DP0 pid=7849) File "", line 1360, in _find_and_load 2025-10-10T01:51:49.6833192Z (EngineCore_DP0 pid=7849) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:49.6833780Z (EngineCore_DP0 pid=7849) File "", line 935, in _load_unlocked 2025-10-10T01:51:49.6834350Z (EngineCore_DP0 pid=7849) File "", line 999, in exec_module 2025-10-10T01:51:49.6834947Z (EngineCore_DP0 pid=7849) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:49.6835682Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:49.6836409Z (EngineCore_DP0 pid=7849) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:49.6837139Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:49.6837908Z (EngineCore_DP0 pid=7849) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:49.6838726Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:49.6839483Z (EngineCore_DP0 pid=7849) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:49.6840273Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:49.6841090Z (EngineCore_DP0 pid=7849) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:49.6841571Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6842264Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:49.6843001Z (EngineCore_DP0 pid=7849) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:49.6843468Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6844220Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:49.6844919Z (EngineCore_DP0 pid=7849) return _is_fa2_supported(device)[0] 2025-10-10T01:51:49.6845337Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6846051Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:49.6846777Z (EngineCore_DP0 pid=7849) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:49.6847246Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6847924Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:49.6848559Z (EngineCore_DP0 pid=7849) prop = get_device_properties(device) 2025-10-10T01:51:49.6848976Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:49.6849632Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:49.6850294Z (EngineCore_DP0 pid=7849) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:49.6850688Z (EngineCore_DP0 pid=7849) ^^^^^^^^^^^^ 2025-10-10T01:51:49.6851320Z (EngineCore_DP0 pid=7849) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:49.6851902Z (EngineCore_DP0 pid=7849) raise RuntimeError( 2025-10-10T01:51:49.6852585Z (EngineCore_DP0 pid=7849) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:50.0843352Z FAILED 2025-10-10T01:51:50.0974363Z models/test_initialization.py::test_can_initialize_large_subset[JAISLMHeadModel] Fork a new process to run a test 7853 2025-10-10T01:51:50.0985495Z Fork a new process to run a test 0 2025-10-10T01:51:50.1261505Z INFO 10-10 01:51:50 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='JAISLMHeadModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'inceptionai/jais-13b-chat'} 2025-10-10T01:51:50.2867955Z 2025-10-10T01:51:50.2869065Z config.json: 0% 0.00/1.26k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:51:58.3855628Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] EngineCore failed to start. 2025-10-10T01:51:58.3856366Z 2025-10-10T01:51:58.3856766Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] Traceback (most recent call last): 2025-10-10T01:51:58.3857146Z 2025-10-10T01:51:58.3857815Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:58.3858400Z 2025-10-10T01:51:58.3858791Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:58.3859172Z 2025-10-10T01:51:58.3859533Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3859886Z 2025-10-10T01:51:58.3860509Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:58.3861129Z 2025-10-10T01:51:58.3861797Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:58.3862226Z 2025-10-10T01:51:58.3862857Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:58.3863420Z 2025-10-10T01:51:58.3863821Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:58.3864128Z 2025-10-10T01:51:58.3864441Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3864846Z 2025-10-10T01:51:58.3865502Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:58.3866266Z 2025-10-10T01:51:58.3866617Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] self._init_executor() 2025-10-10T01:51:58.3866890Z 2025-10-10T01:51:58.3867708Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:58.3868218Z 2025-10-10T01:51:58.3868704Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:58.3869030Z 2025-10-10T01:51:58.3869584Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:58.3870080Z 2025-10-10T01:51:58.3870544Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:58.3870885Z 2025-10-10T01:51:58.3871198Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3871482Z 2025-10-10T01:51:58.3871990Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:58.3872463Z 2025-10-10T01:51:58.3872817Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:51:58.3873136Z 2025-10-10T01:51:58.3873449Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3873775Z 2025-10-10T01:51:58.3874413Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:58.3875033Z 2025-10-10T01:51:58.3875381Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:58.3875691Z 2025-10-10T01:51:58.3875964Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3876227Z 2025-10-10T01:51:58.3876755Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:58.3877235Z 2025-10-10T01:51:58.3877541Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:51:58.3877836Z 2025-10-10T01:51:58.3878107Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3878389Z 2025-10-10T01:51:58.3878868Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:58.3879400Z 2025-10-10T01:51:58.3879747Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:58.3880076Z 2025-10-10T01:51:58.3880368Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3880648Z 2025-10-10T01:51:58.3880994Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:51:58.3881321Z 2025-10-10T01:51:58.3881675Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:51:58.3882006Z 2025-10-10T01:51:58.3882382Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:58.3882735Z 2025-10-10T01:51:58.3883107Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:51:58.3883499Z 2025-10-10T01:51:58.3883928Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:51:58.3884343Z 2025-10-10T01:51:58.3884779Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:58.3885158Z 2025-10-10T01:51:58.3885710Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:58.3886180Z 2025-10-10T01:51:58.3886531Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:58.3886861Z 2025-10-10T01:51:58.3887387Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:58.3887860Z 2025-10-10T01:51:58.3888275Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:58.3888628Z 2025-10-10T01:51:58.3889185Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:58.3889737Z 2025-10-10T01:51:58.3890037Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:51:58.3890327Z 2025-10-10T01:51:58.3890951Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:58.3891522Z 2025-10-10T01:51:58.3891884Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:58.3892237Z 2025-10-10T01:51:58.3892487Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3892765Z 2025-10-10T01:51:58.3893324Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:58.3893851Z 2025-10-10T01:51:58.3894183Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:51:58.3894473Z 2025-10-10T01:51:58.3894748Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3895015Z 2025-10-10T01:51:58.3895612Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:58.3896364Z 2025-10-10T01:51:58.3896653Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:51:58.3896931Z 2025-10-10T01:51:58.3897188Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3897457Z 2025-10-10T01:51:58.3898033Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:58.3898558Z 2025-10-10T01:51:58.3898889Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:58.3899209Z 2025-10-10T01:51:58.3899481Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3899762Z 2025-10-10T01:51:58.3900297Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:58.3900781Z 2025-10-10T01:51:58.3901149Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:51:58.3901448Z 2025-10-10T01:51:58.3901722Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3901986Z 2025-10-10T01:51:58.3902518Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:58.3903003Z 2025-10-10T01:51:58.3903310Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:51:58.3903680Z 2025-10-10T01:51:58.3903906Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:51:58.3904150Z 2025-10-10T01:51:58.3904631Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:58.3905172Z 2025-10-10T01:51:58.3905421Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] raise RuntimeError( 2025-10-10T01:51:58.3905684Z 2025-10-10T01:51:58.3906246Z (EngineCore_DP0 pid=7911) ERROR 10-10 01:51:58 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:58.3906933Z (EngineCore_DP0 pid=7911) Process EngineCore_DP0: 2025-10-10T01:51:58.3907328Z (EngineCore_DP0 pid=7911) Traceback (most recent call last): 2025-10-10T01:51:58.3907951Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:51:58.3908494Z (EngineCore_DP0 pid=7911) self.run() 2025-10-10T01:51:58.3909025Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:51:58.3909663Z (EngineCore_DP0 pid=7911) self._target(*self._args, **self._kwargs) 2025-10-10T01:51:58.3910338Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:51:58.3910883Z (EngineCore_DP0 pid=7911) raise e 2025-10-10T01:51:58.3911463Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:51:58.3912096Z (EngineCore_DP0 pid=7911) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:51:58.3912531Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3913150Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:51:58.3913790Z (EngineCore_DP0 pid=7911) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:51:58.3914443Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:51:58.3915058Z (EngineCore_DP0 pid=7911) self.model_executor = executor_class(vllm_config) 2025-10-10T01:51:58.3915519Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3916167Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:51:58.3916748Z (EngineCore_DP0 pid=7911) self._init_executor() 2025-10-10T01:51:58.3917448Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:51:58.3918136Z (EngineCore_DP0 pid=7911) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:51:58.3918863Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:51:58.3919630Z (EngineCore_DP0 pid=7911) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:51:58.3920122Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3920830Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:51:58.3921404Z (EngineCore_DP0 pid=7911) return func(*args, **kwargs) 2025-10-10T01:51:58.3921788Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3922472Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:51:58.3923115Z (EngineCore_DP0 pid=7911) worker_class = resolve_obj_by_qualname( 2025-10-10T01:51:58.3923538Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3924196Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:51:58.3924845Z (EngineCore_DP0 pid=7911) module = importlib.import_module(module_name) 2025-10-10T01:51:58.3925289Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3925871Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:51:58.3926496Z (EngineCore_DP0 pid=7911) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:51:58.3927029Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3927533Z (EngineCore_DP0 pid=7911) File "", line 1387, in _gcd_import 2025-10-10T01:51:58.3928085Z (EngineCore_DP0 pid=7911) File "", line 1360, in _find_and_load 2025-10-10T01:51:58.3928686Z (EngineCore_DP0 pid=7911) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:51:58.3929266Z (EngineCore_DP0 pid=7911) File "", line 935, in _load_unlocked 2025-10-10T01:51:58.3929828Z (EngineCore_DP0 pid=7911) File "", line 999, in exec_module 2025-10-10T01:51:58.3930433Z (EngineCore_DP0 pid=7911) File "", line 488, in _call_with_frames_removed 2025-10-10T01:51:58.3931172Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:51:58.3931849Z (EngineCore_DP0 pid=7911) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:51:58.3932575Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:51:58.3933367Z (EngineCore_DP0 pid=7911) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:51:58.3934285Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:51:58.3935078Z (EngineCore_DP0 pid=7911) class FlashAttentionMetadataBuilder( 2025-10-10T01:51:58.3936010Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:51:58.3936824Z (EngineCore_DP0 pid=7911) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:51:58.3937299Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3937984Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:51:58.3938725Z (EngineCore_DP0 pid=7911) if not is_fa_version_supported(fa_version): 2025-10-10T01:51:58.3939165Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3939911Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:51:58.3940652Z (EngineCore_DP0 pid=7911) return _is_fa2_supported(device)[0] 2025-10-10T01:51:58.3941062Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3941761Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:51:58.3942471Z (EngineCore_DP0 pid=7911) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:51:58.3942924Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3943592Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:51:58.3944227Z (EngineCore_DP0 pid=7911) prop = get_device_properties(device) 2025-10-10T01:51:58.3944657Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:51:58.3945370Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:51:58.3946041Z (EngineCore_DP0 pid=7911) _lazy_init() # will define _get_device_properties 2025-10-10T01:51:58.3946439Z (EngineCore_DP0 pid=7911) ^^^^^^^^^^^^ 2025-10-10T01:51:58.3947015Z (EngineCore_DP0 pid=7911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:51:58.3947570Z (EngineCore_DP0 pid=7911) raise RuntimeError( 2025-10-10T01:51:58.3948262Z (EngineCore_DP0 pid=7911) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:51:58.7927689Z FAILED 2025-10-10T01:51:58.8057154Z models/test_initialization.py::test_can_initialize_large_subset[MiniCPM3ForCausalLM] Fork a new process to run a test 7915 2025-10-10T01:51:58.8069467Z Fork a new process to run a test 0 2025-10-10T01:51:58.8344190Z INFO 10-10 01:51:58 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiniCPM3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'openbmb/MiniCPM3-4B'} 2025-10-10T01:51:58.9617637Z 2025-10-10T01:51:58.9619991Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:51:58.9620292Z config.json: 1.93kB [00:00, 10.2MB/s] 2025-10-10T01:51:59.0328185Z 2025-10-10T01:51:59.0330424Z configuration_minicpm.py: 0.00B [00:00, ?B/s] 2025-10-10T01:51:59.0330816Z configuration_minicpm.py: 9.23kB [00:00, 45.6MB/s] 2025-10-10T01:51:59.0416808Z A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM3-4B: 2025-10-10T01:51:59.0417371Z - configuration_minicpm.py 2025-10-10T01:51:59.0417998Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:51:59.0758834Z INFO 10-10 01:51:59 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:52:05.8510527Z INFO 10-10 01:52:05 [model.py:551] Resolved architecture: MiniCPM3ForCausalLM 2025-10-10T01:52:05.8511199Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:52:05.8765912Z INFO 10-10 01:52:05 [model.py:1545] Using max model len 32768 2025-10-10T01:52:05.8767618Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:52:05.9424774Z INFO 10-10 01:52:05 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:52:06.0443036Z 2025-10-10T01:52:06.0445573Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:06.0446892Z tokenizer_config.json: 10.4kB [00:00, 45.5MB/s] 2025-10-10T01:52:06.1613131Z 2025-10-10T01:52:06.5056609Z tokenizer.model: 0% 0.00/1.18M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:07.1138039Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:07.1138564Z 2025-10-10T01:52:07.1138959Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:07.1139602Z 2025-10-10T01:52:07.1140292Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:07.1140892Z 2025-10-10T01:52:07.1141292Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:07.1141782Z 2025-10-10T01:52:07.1142075Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1142350Z 2025-10-10T01:52:07.1142829Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:07.1143267Z 2025-10-10T01:52:07.1143599Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:07.1143912Z 2025-10-10T01:52:07.1144506Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:07.1144949Z 2025-10-10T01:52:07.1145285Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:07.1145595Z 2025-10-10T01:52:07.1145970Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1146257Z 2025-10-10T01:52:07.1146763Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:07.1147232Z 2025-10-10T01:52:07.1147478Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] self._init_executor() 2025-10-10T01:52:07.1147864Z 2025-10-10T01:52:07.1148536Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:07.1149073Z 2025-10-10T01:52:07.1149503Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:07.1150051Z 2025-10-10T01:52:07.1151057Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:07.1151787Z 2025-10-10T01:52:07.1152226Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:07.1152726Z 2025-10-10T01:52:07.1153050Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1153386Z 2025-10-10T01:52:07.1153970Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:07.1154881Z 2025-10-10T01:52:07.1155368Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:07.1155858Z 2025-10-10T01:52:07.1156294Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1156720Z 2025-10-10T01:52:07.1157562Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:07.1158313Z 2025-10-10T01:52:07.1158831Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:07.1159417Z 2025-10-10T01:52:07.1159844Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1160268Z 2025-10-10T01:52:07.1161101Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:07.1161927Z 2025-10-10T01:52:07.1162403Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:07.1162877Z 2025-10-10T01:52:07.1163315Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1163753Z 2025-10-10T01:52:07.1164460Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:07.1165163Z 2025-10-10T01:52:07.1165741Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:07.1166302Z 2025-10-10T01:52:07.1166801Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1167326Z 2025-10-10T01:52:07.1167944Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:07.1168535Z 2025-10-10T01:52:07.1169177Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:07.1169790Z 2025-10-10T01:52:07.1170473Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:07.1171097Z 2025-10-10T01:52:07.1171695Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:07.1172270Z 2025-10-10T01:52:07.1172896Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:07.1173504Z 2025-10-10T01:52:07.1174149Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:07.1174774Z 2025-10-10T01:52:07.1175654Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:07.1176469Z 2025-10-10T01:52:07.1177087Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:07.1177673Z 2025-10-10T01:52:07.1178610Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:07.1179574Z 2025-10-10T01:52:07.1180259Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:07.1180897Z 2025-10-10T01:52:07.1181873Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:07.1182754Z 2025-10-10T01:52:07.1183250Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:07.1183826Z 2025-10-10T01:52:07.1184915Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:07.1185917Z 2025-10-10T01:52:07.1186545Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:07.1187246Z 2025-10-10T01:52:07.1187674Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1188100Z 2025-10-10T01:52:07.1189054Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:07.1189990Z 2025-10-10T01:52:07.1190539Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:07.1191070Z 2025-10-10T01:52:07.1191566Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1192061Z 2025-10-10T01:52:07.1193245Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:07.1194244Z 2025-10-10T01:52:07.1194710Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:07.1195179Z 2025-10-10T01:52:07.1195599Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1196039Z 2025-10-10T01:52:07.1197215Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:07.1198158Z 2025-10-10T01:52:07.1198694Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:07.1199283Z 2025-10-10T01:52:07.1199765Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1200240Z 2025-10-10T01:52:07.1201138Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:07.1201959Z 2025-10-10T01:52:07.1202426Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:07.1202915Z 2025-10-10T01:52:07.1203376Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1203858Z 2025-10-10T01:52:07.1204843Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:07.1205711Z 2025-10-10T01:52:07.1206337Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:07.1206856Z 2025-10-10T01:52:07.1207234Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:07.1207642Z 2025-10-10T01:52:07.1208471Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:07.1209243Z 2025-10-10T01:52:07.1209652Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] raise RuntimeError( 2025-10-10T01:52:07.1210181Z 2025-10-10T01:52:07.1211132Z (EngineCore_DP0 pid=7993) ERROR 10-10 01:52:07 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:07.1212337Z (EngineCore_DP0 pid=7993) Process EngineCore_DP0: 2025-10-10T01:52:07.1213089Z (EngineCore_DP0 pid=7993) Traceback (most recent call last): 2025-10-10T01:52:07.1214093Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:07.1215003Z (EngineCore_DP0 pid=7993) self.run() 2025-10-10T01:52:07.1215959Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:07.1216988Z (EngineCore_DP0 pid=7993) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:07.1218156Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:07.1219076Z (EngineCore_DP0 pid=7993) raise e 2025-10-10T01:52:07.1220098Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:07.1221248Z (EngineCore_DP0 pid=7993) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:07.1221997Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1223033Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:07.1224136Z (EngineCore_DP0 pid=7993) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:07.1225273Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:07.1226341Z (EngineCore_DP0 pid=7993) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:07.1227141Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1228348Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:07.1229434Z (EngineCore_DP0 pid=7993) self._init_executor() 2025-10-10T01:52:07.1230501Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:07.1231645Z (EngineCore_DP0 pid=7993) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:07.1232931Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:07.1234181Z (EngineCore_DP0 pid=7993) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:07.1235093Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1236227Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:07.1237272Z (EngineCore_DP0 pid=7993) return func(*args, **kwargs) 2025-10-10T01:52:07.1237922Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1239137Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:07.1240341Z (EngineCore_DP0 pid=7993) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:07.1241115Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1242331Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:07.1243614Z (EngineCore_DP0 pid=7993) module = importlib.import_module(module_name) 2025-10-10T01:52:07.1244424Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1245481Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:07.1246617Z (EngineCore_DP0 pid=7993) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:07.1247510Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1248287Z (EngineCore_DP0 pid=7993) File "", line 1387, in _gcd_import 2025-10-10T01:52:07.1249250Z (EngineCore_DP0 pid=7993) File "", line 1360, in _find_and_load 2025-10-10T01:52:07.1250311Z (EngineCore_DP0 pid=7993) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:07.1251369Z (EngineCore_DP0 pid=7993) File "", line 935, in _load_unlocked 2025-10-10T01:52:07.1252297Z (EngineCore_DP0 pid=7993) File "", line 999, in exec_module 2025-10-10T01:52:07.1253273Z (EngineCore_DP0 pid=7993) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:07.1254473Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:07.1255565Z (EngineCore_DP0 pid=7993) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:07.1256740Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:07.1257966Z (EngineCore_DP0 pid=7993) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:07.1259336Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:07.1260560Z (EngineCore_DP0 pid=7993) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:07.1261963Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:07.1263323Z (EngineCore_DP0 pid=7993) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:07.1264096Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1265289Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:07.1266412Z (EngineCore_DP0 pid=7993) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:07.1267117Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1268310Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:07.1269458Z (EngineCore_DP0 pid=7993) return _is_fa2_supported(device)[0] 2025-10-10T01:52:07.1270167Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1271378Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:07.1272555Z (EngineCore_DP0 pid=7993) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:07.1273455Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1274680Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:07.1275750Z (EngineCore_DP0 pid=7993) prop = get_device_properties(device) 2025-10-10T01:52:07.1276414Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:07.1277482Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:07.1278556Z (EngineCore_DP0 pid=7993) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:07.1279228Z (EngineCore_DP0 pid=7993) ^^^^^^^^^^^^ 2025-10-10T01:52:07.1280161Z (EngineCore_DP0 pid=7993) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:07.1281142Z (EngineCore_DP0 pid=7993) raise RuntimeError( 2025-10-10T01:52:07.1282364Z (EngineCore_DP0 pid=7993) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:07.5241750Z FAILED 2025-10-10T01:52:07.5372007Z models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEEmbeddingModel] Fork a new process to run a test 7997 2025-10-10T01:52:07.5384686Z Fork a new process to run a test 0 2025-10-10T01:52:07.5388824Z `transformers==4.56.2` installed, but `transformers>=4.57.0.dev0` is required to run this model. 2025-10-10T01:52:07.8448258Z PASSED 2025-10-10T01:52:07.8578996Z models/test_initialization.py::test_can_initialize_large_subset[MistralModel] Fork a new process to run a test 7998 2025-10-10T01:52:07.8590549Z Fork a new process to run a test 0 2025-10-10T01:52:07.8864326Z INFO 10-10 01:52:07 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MistralModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'intfloat/e5-mistral-7b-instruct'} 2025-10-10T01:52:07.9645425Z 2025-10-10T01:52:07.9646505Z config.json: 0% 0.00/629 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:09.3908858Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:09.3909621Z 2025-10-10T01:52:09.3910025Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:09.3910390Z 2025-10-10T01:52:09.3911053Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:09.3911651Z 2025-10-10T01:52:09.3912033Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:09.3912416Z 2025-10-10T01:52:09.3912761Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3913100Z 2025-10-10T01:52:09.3913714Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:09.3914167Z 2025-10-10T01:52:09.3914618Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:09.3915038Z 2025-10-10T01:52:09.3915534Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:09.3915967Z 2025-10-10T01:52:09.3916281Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:09.3916592Z 2025-10-10T01:52:09.3916896Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3917171Z 2025-10-10T01:52:09.3917681Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:09.3918215Z 2025-10-10T01:52:09.3918570Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] self._init_executor() 2025-10-10T01:52:09.3919091Z 2025-10-10T01:52:09.3919832Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:09.3920577Z 2025-10-10T01:52:09.3921033Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:09.3921564Z 2025-10-10T01:52:09.3922265Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:09.3922892Z 2025-10-10T01:52:09.3923352Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:09.3923697Z 2025-10-10T01:52:09.3924007Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3924285Z 2025-10-10T01:52:09.3924772Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:09.3925218Z 2025-10-10T01:52:09.3925482Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:09.3925823Z 2025-10-10T01:52:09.3926073Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3926332Z 2025-10-10T01:52:09.3926840Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:09.3927358Z 2025-10-10T01:52:09.3927648Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:09.3927935Z 2025-10-10T01:52:09.3928201Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3928468Z 2025-10-10T01:52:09.3928993Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:09.3929471Z 2025-10-10T01:52:09.3929775Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:09.3930066Z 2025-10-10T01:52:09.3930343Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3930612Z 2025-10-10T01:52:09.3931097Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:09.3931505Z 2025-10-10T01:52:09.3931847Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:09.3932190Z 2025-10-10T01:52:09.3932474Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3932759Z 2025-10-10T01:52:09.3933100Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:09.3933439Z 2025-10-10T01:52:09.3933782Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:09.3934113Z 2025-10-10T01:52:09.3934487Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:09.3934841Z 2025-10-10T01:52:09.3935188Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:09.3935513Z 2025-10-10T01:52:09.3935876Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:09.3936224Z 2025-10-10T01:52:09.3936601Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:09.3936956Z 2025-10-10T01:52:09.3937487Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:09.3937970Z 2025-10-10T01:52:09.3938320Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:09.3938656Z 2025-10-10T01:52:09.3939171Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:09.3939649Z 2025-10-10T01:52:09.3940038Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:09.3940448Z 2025-10-10T01:52:09.3941015Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:09.3941565Z 2025-10-10T01:52:09.3941865Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:09.3942156Z 2025-10-10T01:52:09.3942783Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:09.3943356Z 2025-10-10T01:52:09.3943724Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:09.3944072Z 2025-10-10T01:52:09.3944324Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3944584Z 2025-10-10T01:52:09.3945144Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:09.3945669Z 2025-10-10T01:52:09.3946002Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:09.3946315Z 2025-10-10T01:52:09.3946593Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3946863Z 2025-10-10T01:52:09.3947456Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:09.3948001Z 2025-10-10T01:52:09.3948285Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:09.3948564Z 2025-10-10T01:52:09.3948820Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3949089Z 2025-10-10T01:52:09.3949660Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:09.3950191Z 2025-10-10T01:52:09.3950502Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:09.3950813Z 2025-10-10T01:52:09.3951082Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3951351Z 2025-10-10T01:52:09.3951883Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:09.3952362Z 2025-10-10T01:52:09.3952692Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:09.3952976Z 2025-10-10T01:52:09.3953247Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3953511Z 2025-10-10T01:52:09.3954020Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:09.3954518Z 2025-10-10T01:52:09.3954823Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:09.3955163Z 2025-10-10T01:52:09.3955386Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:09.3955625Z 2025-10-10T01:52:09.3956108Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:09.3956601Z 2025-10-10T01:52:09.3956847Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] raise RuntimeError( 2025-10-10T01:52:09.3957098Z 2025-10-10T01:52:09.3957679Z (EngineCore_DP0 pid=8026) ERROR 10-10 01:52:09 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:09.3958363Z (EngineCore_DP0 pid=8026) Process EngineCore_DP0: 2025-10-10T01:52:09.3958745Z (EngineCore_DP0 pid=8026) Traceback (most recent call last): 2025-10-10T01:52:09.3959503Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:09.3960010Z (EngineCore_DP0 pid=8026) self.run() 2025-10-10T01:52:09.3960544Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:09.3961173Z (EngineCore_DP0 pid=8026) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:09.3961843Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:09.3962386Z (EngineCore_DP0 pid=8026) raise e 2025-10-10T01:52:09.3962966Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:09.3963590Z (EngineCore_DP0 pid=8026) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:09.3964037Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3964671Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:09.3965310Z (EngineCore_DP0 pid=8026) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:09.3965958Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:09.3966582Z (EngineCore_DP0 pid=8026) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:09.3967025Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3967670Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:09.3968250Z (EngineCore_DP0 pid=8026) self._init_executor() 2025-10-10T01:52:09.3968901Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:09.3969652Z (EngineCore_DP0 pid=8026) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:09.3970379Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:09.3971084Z (EngineCore_DP0 pid=8026) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:09.3971572Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3972206Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:09.3972833Z (EngineCore_DP0 pid=8026) return func(*args, **kwargs) 2025-10-10T01:52:09.3973211Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3973852Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:09.3974520Z (EngineCore_DP0 pid=8026) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:09.3974933Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3975596Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:09.3976243Z (EngineCore_DP0 pid=8026) module = importlib.import_module(module_name) 2025-10-10T01:52:09.3976686Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3977276Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:09.3977905Z (EngineCore_DP0 pid=8026) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:09.3978432Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3978928Z (EngineCore_DP0 pid=8026) File "", line 1387, in _gcd_import 2025-10-10T01:52:09.3979489Z (EngineCore_DP0 pid=8026) File "", line 1360, in _find_and_load 2025-10-10T01:52:09.3980079Z (EngineCore_DP0 pid=8026) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:09.3980655Z (EngineCore_DP0 pid=8026) File "", line 935, in _load_unlocked 2025-10-10T01:52:09.3981220Z (EngineCore_DP0 pid=8026) File "", line 999, in exec_module 2025-10-10T01:52:09.3981816Z (EngineCore_DP0 pid=8026) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:09.3982544Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:09.3983211Z (EngineCore_DP0 pid=8026) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:09.3983918Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:09.3984621Z (EngineCore_DP0 pid=8026) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:09.3985392Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:09.3986053Z (EngineCore_DP0 pid=8026) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:09.3986874Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:09.3987689Z (EngineCore_DP0 pid=8026) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:09.3988157Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3988832Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:09.3989522Z (EngineCore_DP0 pid=8026) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:09.3990006Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3990737Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:09.3991465Z (EngineCore_DP0 pid=8026) return _is_fa2_supported(device)[0] 2025-10-10T01:52:09.3991863Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3992572Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:09.3993292Z (EngineCore_DP0 pid=8026) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:09.3993739Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3994402Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:09.3995022Z (EngineCore_DP0 pid=8026) prop = get_device_properties(device) 2025-10-10T01:52:09.3995434Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:09.3996367Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:09.3997090Z (EngineCore_DP0 pid=8026) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:09.3997482Z (EngineCore_DP0 pid=8026) ^^^^^^^^^^^^ 2025-10-10T01:52:09.3998055Z (EngineCore_DP0 pid=8026) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:09.3998617Z (EngineCore_DP0 pid=8026) raise RuntimeError( 2025-10-10T01:52:09.3999352Z (EngineCore_DP0 pid=8026) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:09.7931804Z FAILED 2025-10-10T01:52:09.8061296Z models/test_initialization.py::test_can_initialize_large_subset[DeepseekForCausalLM] Fork a new process to run a test 8030 2025-10-10T01:52:09.8072714Z Fork a new process to run a test 0 2025-10-10T01:52:09.8351550Z INFO 10-10 01:52:09 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DeepseekForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'deepseek-ai/deepseek-llm-7b-chat'} 2025-10-10T01:52:10.0044912Z 2025-10-10T01:52:10.0045940Z config.json: 0% 0.00/594 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:11.1517052Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:11.1517709Z 2025-10-10T01:52:11.1518352Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:11.1518740Z 2025-10-10T01:52:11.1519568Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:11.1520159Z 2025-10-10T01:52:11.1520551Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:11.1520934Z 2025-10-10T01:52:11.1521278Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1521612Z 2025-10-10T01:52:11.1522460Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:11.1523028Z 2025-10-10T01:52:11.1523451Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:11.1523850Z 2025-10-10T01:52:11.1524442Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:11.1524990Z 2025-10-10T01:52:11.1525367Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:11.1525757Z 2025-10-10T01:52:11.1526031Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1526303Z 2025-10-10T01:52:11.1526830Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:11.1527382Z 2025-10-10T01:52:11.1527641Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] self._init_executor() 2025-10-10T01:52:11.1527895Z 2025-10-10T01:52:11.1528437Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:11.1529082Z 2025-10-10T01:52:11.1529421Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:11.1529808Z 2025-10-10T01:52:11.1530451Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:11.1531084Z 2025-10-10T01:52:11.1531455Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:11.1532043Z 2025-10-10T01:52:11.1532401Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1532791Z 2025-10-10T01:52:11.1533350Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:11.1533862Z 2025-10-10T01:52:11.1534182Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:11.1534522Z 2025-10-10T01:52:11.1534786Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1535129Z 2025-10-10T01:52:11.1535702Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:11.1536306Z 2025-10-10T01:52:11.1536610Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:11.1537018Z 2025-10-10T01:52:11.1537292Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1537572Z 2025-10-10T01:52:11.1538284Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:11.1538895Z 2025-10-10T01:52:11.1539217Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:11.1539634Z 2025-10-10T01:52:11.1540012Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1540403Z 2025-10-10T01:52:11.1540872Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:11.1541411Z 2025-10-10T01:52:11.1541760Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:11.1542212Z 2025-10-10T01:52:11.1542511Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1542925Z 2025-10-10T01:52:11.1543348Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:11.1543699Z 2025-10-10T01:52:11.1544167Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:11.1544588Z 2025-10-10T01:52:11.1545105Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:11.1545522Z 2025-10-10T01:52:11.1545952Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:11.1546289Z 2025-10-10T01:52:11.1546787Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:11.1547146Z 2025-10-10T01:52:11.1547654Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:11.1548024Z 2025-10-10T01:52:11.1548705Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:11.1549302Z 2025-10-10T01:52:11.1549672Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:11.1550151Z 2025-10-10T01:52:11.1550746Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:11.1551298Z 2025-10-10T01:52:11.1551811Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:11.1552174Z 2025-10-10T01:52:11.1552871Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:11.1553489Z 2025-10-10T01:52:11.1553806Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:11.1554109Z 2025-10-10T01:52:11.1554861Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:11.1555562Z 2025-10-10T01:52:11.1556050Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:11.1556416Z 2025-10-10T01:52:11.1556676Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1557062Z 2025-10-10T01:52:11.1557774Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:11.1558350Z 2025-10-10T01:52:11.1558789Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:11.1559163Z 2025-10-10T01:52:11.1559443Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1559843Z 2025-10-10T01:52:11.1560464Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:11.1561151Z 2025-10-10T01:52:11.1561515Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:11.1561797Z 2025-10-10T01:52:11.1562063Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1562375Z 2025-10-10T01:52:11.1562969Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:11.1563643Z 2025-10-10T01:52:11.1563972Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:11.1564283Z 2025-10-10T01:52:11.1564551Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1564827Z 2025-10-10T01:52:11.1565345Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:11.1565827Z 2025-10-10T01:52:11.1566109Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:11.1566385Z 2025-10-10T01:52:11.1566702Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1566968Z 2025-10-10T01:52:11.1567638Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:11.1568118Z 2025-10-10T01:52:11.1568436Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:11.1568738Z 2025-10-10T01:52:11.1568960Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:11.1569203Z 2025-10-10T01:52:11.1569682Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:11.1570135Z 2025-10-10T01:52:11.1570381Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] raise RuntimeError( 2025-10-10T01:52:11.1570638Z 2025-10-10T01:52:11.1571196Z (EngineCore_DP0 pid=8038) ERROR 10-10 01:52:11 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:11.1571879Z (EngineCore_DP0 pid=8038) Process EngineCore_DP0: 2025-10-10T01:52:11.1572272Z (EngineCore_DP0 pid=8038) Traceback (most recent call last): 2025-10-10T01:52:11.1572872Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:11.1573398Z (EngineCore_DP0 pid=8038) self.run() 2025-10-10T01:52:11.1573973Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:11.1574696Z (EngineCore_DP0 pid=8038) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:11.1582318Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:11.1582926Z (EngineCore_DP0 pid=8038) raise e 2025-10-10T01:52:11.1583534Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:11.1584276Z (EngineCore_DP0 pid=8038) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:11.1584723Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1585356Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:11.1586063Z (EngineCore_DP0 pid=8038) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:11.1586741Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:11.1587367Z (EngineCore_DP0 pid=8038) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:11.1587837Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1588705Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:11.1589304Z (EngineCore_DP0 pid=8038) self._init_executor() 2025-10-10T01:52:11.1589964Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:11.1590709Z (EngineCore_DP0 pid=8038) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:11.1591439Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:11.1592152Z (EngineCore_DP0 pid=8038) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:11.1592647Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1593290Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:11.1593873Z (EngineCore_DP0 pid=8038) return func(*args, **kwargs) 2025-10-10T01:52:11.1594261Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1594906Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:11.1595540Z (EngineCore_DP0 pid=8038) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:11.1595968Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1596880Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:11.1597543Z (EngineCore_DP0 pid=8038) module = importlib.import_module(module_name) 2025-10-10T01:52:11.1597993Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1598573Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:11.1599369Z (EngineCore_DP0 pid=8038) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:11.1599871Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1600367Z (EngineCore_DP0 pid=8038) File "", line 1387, in _gcd_import 2025-10-10T01:52:11.1600922Z (EngineCore_DP0 pid=8038) File "", line 1360, in _find_and_load 2025-10-10T01:52:11.1601511Z (EngineCore_DP0 pid=8038) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:11.1602165Z (EngineCore_DP0 pid=8038) File "", line 935, in _load_unlocked 2025-10-10T01:52:11.1602728Z (EngineCore_DP0 pid=8038) File "", line 999, in exec_module 2025-10-10T01:52:11.1603327Z (EngineCore_DP0 pid=8038) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:11.1604120Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:11.1604802Z (EngineCore_DP0 pid=8038) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:11.1605523Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:11.1606241Z (EngineCore_DP0 pid=8038) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:11.1607021Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:11.1607708Z (EngineCore_DP0 pid=8038) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:11.1608566Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:11.1609390Z (EngineCore_DP0 pid=8038) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:11.1609866Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1610552Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:11.1611233Z (EngineCore_DP0 pid=8038) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:11.1611669Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1612413Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:11.1613120Z (EngineCore_DP0 pid=8038) return _is_fa2_supported(device)[0] 2025-10-10T01:52:11.1613528Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1614269Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:11.1615158Z (EngineCore_DP0 pid=8038) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:11.1615617Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1616293Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:11.1616935Z (EngineCore_DP0 pid=8038) prop = get_device_properties(device) 2025-10-10T01:52:11.1617408Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:11.1618075Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:11.1618737Z (EngineCore_DP0 pid=8038) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:11.1619133Z (EngineCore_DP0 pid=8038) ^^^^^^^^^^^^ 2025-10-10T01:52:11.1619713Z (EngineCore_DP0 pid=8038) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:11.1620329Z (EngineCore_DP0 pid=8038) raise RuntimeError( 2025-10-10T01:52:11.1621015Z (EngineCore_DP0 pid=8038) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:11.5592102Z FAILED 2025-10-10T01:52:11.5721710Z models/test_initialization.py::test_can_initialize_large_subset[GPTNeoXForCausalLM] Fork a new process to run a test 8042 2025-10-10T01:52:11.5733280Z Fork a new process to run a test 0 2025-10-10T01:52:11.6016872Z INFO 10-10 01:52:11 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GPTNeoXForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'EleutherAI/pythia-70m'} 2025-10-10T01:52:11.7111347Z 2025-10-10T01:52:11.7112589Z config.json: 0% 0.00/567 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:19.4133604Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:19.4134112Z 2025-10-10T01:52:19.4134576Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:19.4134953Z 2025-10-10T01:52:19.4135622Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:19.4136231Z 2025-10-10T01:52:19.4136622Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:19.4137032Z 2025-10-10T01:52:19.4137373Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4137717Z 2025-10-10T01:52:19.4138320Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:19.4139117Z 2025-10-10T01:52:19.4139539Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:19.4139939Z 2025-10-10T01:52:19.4140524Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:19.4141079Z 2025-10-10T01:52:19.4141503Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:19.4141897Z 2025-10-10T01:52:19.4142265Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4142673Z 2025-10-10T01:52:19.4143474Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:19.4144066Z 2025-10-10T01:52:19.4144393Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] self._init_executor() 2025-10-10T01:52:19.4144886Z 2025-10-10T01:52:19.4145749Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:19.4146481Z 2025-10-10T01:52:19.4146847Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:19.4147181Z 2025-10-10T01:52:19.4147732Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:19.4148236Z 2025-10-10T01:52:19.4148710Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:19.4149053Z 2025-10-10T01:52:19.4149364Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4149644Z 2025-10-10T01:52:19.4150137Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:19.4150664Z 2025-10-10T01:52:19.4150930Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:19.4151201Z 2025-10-10T01:52:19.4151461Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4151727Z 2025-10-10T01:52:19.4152344Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:19.4152834Z 2025-10-10T01:52:19.4153122Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:19.4153421Z 2025-10-10T01:52:19.4153699Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4153970Z 2025-10-10T01:52:19.4154506Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:19.4154993Z 2025-10-10T01:52:19.4155297Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:19.4155598Z 2025-10-10T01:52:19.4155885Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4156220Z 2025-10-10T01:52:19.4156686Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:19.4157108Z 2025-10-10T01:52:19.4157457Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:19.4157811Z 2025-10-10T01:52:19.4158101Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4158388Z 2025-10-10T01:52:19.4158728Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:19.4159173Z 2025-10-10T01:52:19.4159534Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:19.4159873Z 2025-10-10T01:52:19.4160249Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:19.4160605Z 2025-10-10T01:52:19.4160950Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:19.4161279Z 2025-10-10T01:52:19.4161642Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:19.4161994Z 2025-10-10T01:52:19.4162362Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:19.4162732Z 2025-10-10T01:52:19.4163282Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:19.4163753Z 2025-10-10T01:52:19.4164110Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:19.4164453Z 2025-10-10T01:52:19.4164977Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:19.4165500Z 2025-10-10T01:52:19.4165871Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:19.4166226Z 2025-10-10T01:52:19.4166794Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:19.4167346Z 2025-10-10T01:52:19.4167650Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:19.4167943Z 2025-10-10T01:52:19.4168582Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:19.4169152Z 2025-10-10T01:52:19.4169520Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:19.4169880Z 2025-10-10T01:52:19.4170133Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4170399Z 2025-10-10T01:52:19.4171010Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:19.4171536Z 2025-10-10T01:52:19.4171830Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:19.4172130Z 2025-10-10T01:52:19.4172397Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4172669Z 2025-10-10T01:52:19.4173269Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:19.4173819Z 2025-10-10T01:52:19.4174104Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:19.4174386Z 2025-10-10T01:52:19.4174666Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4174933Z 2025-10-10T01:52:19.4175516Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:19.4176051Z 2025-10-10T01:52:19.4176366Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:19.4176682Z 2025-10-10T01:52:19.4176952Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4177233Z 2025-10-10T01:52:19.4177750Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:19.4178241Z 2025-10-10T01:52:19.4178585Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:19.4178870Z 2025-10-10T01:52:19.4179136Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4179401Z 2025-10-10T01:52:19.4179921Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:19.4180398Z 2025-10-10T01:52:19.4180769Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:19.4181074Z 2025-10-10T01:52:19.4181296Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:19.4181540Z 2025-10-10T01:52:19.4182032Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:19.4182542Z 2025-10-10T01:52:19.4182787Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] raise RuntimeError( 2025-10-10T01:52:19.4183050Z 2025-10-10T01:52:19.4183612Z (EngineCore_DP0 pid=8100) ERROR 10-10 01:52:19 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:19.4184309Z (EngineCore_DP0 pid=8100) Process EngineCore_DP0: 2025-10-10T01:52:19.4184706Z (EngineCore_DP0 pid=8100) Traceback (most recent call last): 2025-10-10T01:52:19.4185326Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:19.4185849Z (EngineCore_DP0 pid=8100) self.run() 2025-10-10T01:52:19.4186376Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:19.4186978Z (EngineCore_DP0 pid=8100) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:19.4187645Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:19.4188185Z (EngineCore_DP0 pid=8100) raise e 2025-10-10T01:52:19.4188766Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:19.4189388Z (EngineCore_DP0 pid=8100) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:19.4189824Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4190440Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:19.4191082Z (EngineCore_DP0 pid=8100) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:19.4191748Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:19.4192398Z (EngineCore_DP0 pid=8100) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:19.4192850Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4193516Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:19.4194129Z (EngineCore_DP0 pid=8100) self._init_executor() 2025-10-10T01:52:19.4194860Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:19.4195561Z (EngineCore_DP0 pid=8100) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:19.4196839Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:19.4197888Z (EngineCore_DP0 pid=8100) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:19.4198392Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4199244Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:19.4199846Z (EngineCore_DP0 pid=8100) return func(*args, **kwargs) 2025-10-10T01:52:19.4200232Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4200956Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:19.4201591Z (EngineCore_DP0 pid=8100) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:19.4202019Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4202705Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:19.4203365Z (EngineCore_DP0 pid=8100) module = importlib.import_module(module_name) 2025-10-10T01:52:19.4203810Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4204394Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:19.4205024Z (EngineCore_DP0 pid=8100) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:19.4205575Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4206069Z (EngineCore_DP0 pid=8100) File "", line 1387, in _gcd_import 2025-10-10T01:52:19.4206625Z (EngineCore_DP0 pid=8100) File "", line 1360, in _find_and_load 2025-10-10T01:52:19.4207212Z (EngineCore_DP0 pid=8100) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:19.4207805Z (EngineCore_DP0 pid=8100) File "", line 935, in _load_unlocked 2025-10-10T01:52:19.4208373Z (EngineCore_DP0 pid=8100) File "", line 999, in exec_module 2025-10-10T01:52:19.4208967Z (EngineCore_DP0 pid=8100) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:19.4209706Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:19.4210385Z (EngineCore_DP0 pid=8100) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:19.4211122Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:19.4211846Z (EngineCore_DP0 pid=8100) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:19.4212640Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:19.4213310Z (EngineCore_DP0 pid=8100) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:19.4214167Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:19.4214990Z (EngineCore_DP0 pid=8100) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:19.4215470Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4216163Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:19.4216882Z (EngineCore_DP0 pid=8100) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:19.4217317Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4218068Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:19.4218816Z (EngineCore_DP0 pid=8100) return _is_fa2_supported(device)[0] 2025-10-10T01:52:19.4219237Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4219944Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:19.4220654Z (EngineCore_DP0 pid=8100) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:19.4221110Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4221777Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:19.4222409Z (EngineCore_DP0 pid=8100) prop = get_device_properties(device) 2025-10-10T01:52:19.4222828Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:19.4223519Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:19.4224186Z (EngineCore_DP0 pid=8100) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:19.4224578Z (EngineCore_DP0 pid=8100) ^^^^^^^^^^^^ 2025-10-10T01:52:19.4225156Z (EngineCore_DP0 pid=8100) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:19.4225716Z (EngineCore_DP0 pid=8100) raise RuntimeError( 2025-10-10T01:52:19.4226389Z (EngineCore_DP0 pid=8100) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:19.8141001Z FAILED 2025-10-10T01:52:19.8269635Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2VLForConditionalGeneration] Fork a new process to run a test 8104 2025-10-10T01:52:19.8281227Z Fork a new process to run a test 0 2025-10-10T01:52:19.8559287Z INFO 10-10 01:52:19 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2VLForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen2-VL-2B-Instruct'} 2025-10-10T01:52:19.9307365Z 2025-10-10T01:52:19.9309313Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:19.9309620Z config.json: 1.20kB [00:00, 6.51MB/s] 2025-10-10T01:52:20.0487286Z 2025-10-10T01:52:20.0487987Z preprocessor_config.json: 0% 0.00/347 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:21.3937540Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:21.3937958Z 2025-10-10T01:52:21.3938349Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:21.3938705Z 2025-10-10T01:52:21.3939641Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:21.3940256Z 2025-10-10T01:52:21.3940663Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:21.3941047Z 2025-10-10T01:52:21.3941394Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3941741Z 2025-10-10T01:52:21.3942347Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:21.3943423Z 2025-10-10T01:52:21.3943907Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:21.3944327Z 2025-10-10T01:52:21.3944841Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:21.3945391Z 2025-10-10T01:52:21.3945758Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:21.3946323Z 2025-10-10T01:52:21.3946887Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3947426Z 2025-10-10T01:52:21.3948432Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:21.3949216Z 2025-10-10T01:52:21.3949541Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] self._init_executor() 2025-10-10T01:52:21.3949818Z 2025-10-10T01:52:21.3950392Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:21.3950909Z 2025-10-10T01:52:21.3951352Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:21.3951695Z 2025-10-10T01:52:21.3952239Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:21.3952740Z 2025-10-10T01:52:21.3953082Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:21.3953418Z 2025-10-10T01:52:21.3953704Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3953995Z 2025-10-10T01:52:21.3954503Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:21.3954970Z 2025-10-10T01:52:21.3955238Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:21.3955501Z 2025-10-10T01:52:21.3955751Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3956005Z 2025-10-10T01:52:21.3956521Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:21.3956987Z 2025-10-10T01:52:21.3957270Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:21.3957556Z 2025-10-10T01:52:21.3957875Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3958149Z 2025-10-10T01:52:21.3958690Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:21.3959314Z 2025-10-10T01:52:21.3959624Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:21.3959924Z 2025-10-10T01:52:21.3960200Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3960535Z 2025-10-10T01:52:21.3960980Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:21.3961391Z 2025-10-10T01:52:21.3961758Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:21.3962142Z 2025-10-10T01:52:21.3962434Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3962708Z 2025-10-10T01:52:21.3963052Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:21.3963380Z 2025-10-10T01:52:21.3963728Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:21.3964062Z 2025-10-10T01:52:21.3964440Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:21.3964808Z 2025-10-10T01:52:21.3965147Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:21.3965496Z 2025-10-10T01:52:21.3965906Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:21.3966264Z 2025-10-10T01:52:21.3966638Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:21.3966994Z 2025-10-10T01:52:21.3967491Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:21.3967949Z 2025-10-10T01:52:21.3968299Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:21.3968627Z 2025-10-10T01:52:21.3969149Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:21.3969619Z 2025-10-10T01:52:21.3969988Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:21.3970339Z 2025-10-10T01:52:21.3970886Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:21.3971391Z 2025-10-10T01:52:21.3971681Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:21.3971990Z 2025-10-10T01:52:21.3972660Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:21.3973250Z 2025-10-10T01:52:21.3973618Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:21.3973970Z 2025-10-10T01:52:21.3974220Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3974478Z 2025-10-10T01:52:21.3975041Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:21.3975613Z 2025-10-10T01:52:21.3975916Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:21.3976207Z 2025-10-10T01:52:21.3976483Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3976802Z 2025-10-10T01:52:21.3977407Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:21.3977952Z 2025-10-10T01:52:21.3978230Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:21.3978516Z 2025-10-10T01:52:21.3978772Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3979040Z 2025-10-10T01:52:21.3979611Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:21.3980136Z 2025-10-10T01:52:21.3980452Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:21.3980796Z 2025-10-10T01:52:21.3981078Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3981349Z 2025-10-10T01:52:21.3981878Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:21.3982357Z 2025-10-10T01:52:21.3982639Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:21.3982921Z 2025-10-10T01:52:21.3983184Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3983449Z 2025-10-10T01:52:21.3983962Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:21.3984451Z 2025-10-10T01:52:21.3984757Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:21.3985060Z 2025-10-10T01:52:21.3985283Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:21.3985529Z 2025-10-10T01:52:21.3986013Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:21.3986457Z 2025-10-10T01:52:21.3986704Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] raise RuntimeError( 2025-10-10T01:52:21.3986953Z 2025-10-10T01:52:21.3987570Z (EngineCore_DP0 pid=8112) ERROR 10-10 01:52:21 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:21.3988271Z (EngineCore_DP0 pid=8112) Process EngineCore_DP0: 2025-10-10T01:52:21.3988662Z (EngineCore_DP0 pid=8112) Traceback (most recent call last): 2025-10-10T01:52:21.3989262Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:21.3989770Z (EngineCore_DP0 pid=8112) self.run() 2025-10-10T01:52:21.3990286Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:21.3990886Z (EngineCore_DP0 pid=8112) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:21.3991545Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:21.3992154Z (EngineCore_DP0 pid=8112) raise e 2025-10-10T01:52:21.3992746Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:21.3993384Z (EngineCore_DP0 pid=8112) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:21.3993830Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3994451Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:21.3995096Z (EngineCore_DP0 pid=8112) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:21.3995753Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:21.3996581Z (EngineCore_DP0 pid=8112) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:21.3997126Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.3997809Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:21.3998402Z (EngineCore_DP0 pid=8112) self._init_executor() 2025-10-10T01:52:21.3999127Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:21.3999817Z (EngineCore_DP0 pid=8112) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:21.4000535Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:21.4001241Z (EngineCore_DP0 pid=8112) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:21.4001734Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4002374Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:21.4002959Z (EngineCore_DP0 pid=8112) return func(*args, **kwargs) 2025-10-10T01:52:21.4003342Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4003971Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:21.4004596Z (EngineCore_DP0 pid=8112) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:21.4005023Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4005765Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:21.4006428Z (EngineCore_DP0 pid=8112) module = importlib.import_module(module_name) 2025-10-10T01:52:21.4006865Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4007452Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:21.4008065Z (EngineCore_DP0 pid=8112) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:21.4008627Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4009138Z (EngineCore_DP0 pid=8112) File "", line 1387, in _gcd_import 2025-10-10T01:52:21.4009697Z (EngineCore_DP0 pid=8112) File "", line 1360, in _find_and_load 2025-10-10T01:52:21.4010348Z (EngineCore_DP0 pid=8112) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:21.4010918Z (EngineCore_DP0 pid=8112) File "", line 935, in _load_unlocked 2025-10-10T01:52:21.4011500Z (EngineCore_DP0 pid=8112) File "", line 999, in exec_module 2025-10-10T01:52:21.4012105Z (EngineCore_DP0 pid=8112) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:21.4012833Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:21.4013511Z (EngineCore_DP0 pid=8112) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:21.4014239Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:21.4014996Z (EngineCore_DP0 pid=8112) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:21.4015790Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:21.4016454Z (EngineCore_DP0 pid=8112) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:21.4017233Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:21.4018039Z (EngineCore_DP0 pid=8112) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:21.4018504Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4019194Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:21.4019886Z (EngineCore_DP0 pid=8112) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:21.4020318Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4021054Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:21.4021745Z (EngineCore_DP0 pid=8112) return _is_fa2_supported(device)[0] 2025-10-10T01:52:21.4022144Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4022896Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:21.4023616Z (EngineCore_DP0 pid=8112) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:21.4024067Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4024730Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:21.4025355Z (EngineCore_DP0 pid=8112) prop = get_device_properties(device) 2025-10-10T01:52:21.4025788Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:21.4026488Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:21.4027143Z (EngineCore_DP0 pid=8112) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:21.4027577Z (EngineCore_DP0 pid=8112) ^^^^^^^^^^^^ 2025-10-10T01:52:21.4028160Z (EngineCore_DP0 pid=8112) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:21.4028717Z (EngineCore_DP0 pid=8112) raise RuntimeError( 2025-10-10T01:52:21.4029398Z (EngineCore_DP0 pid=8112) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:21.8096501Z FAILED 2025-10-10T01:52:21.8225385Z models/test_initialization.py::test_can_initialize_large_subset[KeyeVL1_5ForConditionalGeneration] Fork a new process to run a test 8116 2025-10-10T01:52:21.8236747Z Fork a new process to run a test 0 2025-10-10T01:52:21.8512733Z INFO 10-10 01:52:21 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='KeyeVL1_5ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Kwai-Keye/Keye-VL-1_5-8B'} 2025-10-10T01:52:21.9755924Z 2025-10-10T01:52:21.9757955Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:21.9758259Z config.json: 2.03kB [00:00, 11.3MB/s] 2025-10-10T01:52:22.0527805Z 2025-10-10T01:52:22.0529682Z configuration_keye_vl_1_5.py: 0.00B [00:00, ?B/s] 2025-10-10T01:52:22.0530094Z configuration_keye_vl_1_5.py: 12.9kB [00:00, 77.2MB/s] 2025-10-10T01:52:22.0620102Z A new version of the following files was downloaded from https://huggingface.co/Kwai-Keye/Keye-VL-1_5-8B: 2025-10-10T01:52:22.0620550Z - configuration_keye_vl_1_5.py 2025-10-10T01:52:22.0621053Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:52:22.2700824Z 2025-10-10T01:52:22.2701302Z preprocessor_config.json: 0% 0.00/458 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:30.6998895Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:30.6999405Z 2025-10-10T01:52:30.6999781Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:30.7000160Z 2025-10-10T01:52:30.7000841Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:30.7001434Z 2025-10-10T01:52:30.7001828Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:30.7002475Z 2025-10-10T01:52:30.7002846Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7003206Z 2025-10-10T01:52:30.7003694Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:30.7004140Z 2025-10-10T01:52:30.7004463Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:30.7004875Z 2025-10-10T01:52:30.7005341Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:30.7005797Z 2025-10-10T01:52:30.7006109Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:30.7006509Z 2025-10-10T01:52:30.7006817Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7007091Z 2025-10-10T01:52:30.7007608Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:30.7008071Z 2025-10-10T01:52:30.7008321Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] self._init_executor() 2025-10-10T01:52:30.7008579Z 2025-10-10T01:52:30.7009112Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:30.7009610Z 2025-10-10T01:52:30.7009926Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:30.7010246Z 2025-10-10T01:52:30.7010873Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:30.7011379Z 2025-10-10T01:52:30.7011725Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:30.7012057Z 2025-10-10T01:52:30.7012341Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7012620Z 2025-10-10T01:52:30.7013117Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:30.7013565Z 2025-10-10T01:52:30.7013843Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:30.7014113Z 2025-10-10T01:52:30.7014371Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7014725Z 2025-10-10T01:52:30.7015601Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:30.7016459Z 2025-10-10T01:52:30.7016949Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:30.7017429Z 2025-10-10T01:52:30.7017906Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7018324Z 2025-10-10T01:52:30.7019218Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:30.7019992Z 2025-10-10T01:52:30.7020469Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:30.7020943Z 2025-10-10T01:52:30.7021367Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7021801Z 2025-10-10T01:52:30.7022487Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:30.7023198Z 2025-10-10T01:52:30.7023729Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:30.7024260Z 2025-10-10T01:52:30.7024721Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7025218Z 2025-10-10T01:52:30.7025798Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:30.7026353Z 2025-10-10T01:52:30.7026948Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:30.7027521Z 2025-10-10T01:52:30.7028159Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:30.7028797Z 2025-10-10T01:52:30.7029413Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:30.7030014Z 2025-10-10T01:52:30.7030689Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:30.7031336Z 2025-10-10T01:52:30.7032046Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:30.7032669Z 2025-10-10T01:52:30.7033506Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:30.7034297Z 2025-10-10T01:52:30.7034887Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:30.7035451Z 2025-10-10T01:52:30.7036367Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:30.7037207Z 2025-10-10T01:52:30.7037878Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:30.7038506Z 2025-10-10T01:52:30.7039597Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:30.7040531Z 2025-10-10T01:52:30.7041089Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:30.7041602Z 2025-10-10T01:52:30.7042733Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:30.7043724Z 2025-10-10T01:52:30.7044404Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:30.7045006Z 2025-10-10T01:52:30.7045438Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7045881Z 2025-10-10T01:52:30.7046850Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:30.7047741Z 2025-10-10T01:52:30.7048236Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:30.7048782Z 2025-10-10T01:52:30.7049240Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7049688Z 2025-10-10T01:52:30.7050697Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:30.7051737Z 2025-10-10T01:52:30.7052250Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:30.7052753Z 2025-10-10T01:52:30.7053216Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7053702Z 2025-10-10T01:52:30.7054752Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:30.7055747Z 2025-10-10T01:52:30.7056281Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:30.7056790Z 2025-10-10T01:52:30.7057375Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7057834Z 2025-10-10T01:52:30.7058795Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:30.7059595Z 2025-10-10T01:52:30.7060077Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:30.7060551Z 2025-10-10T01:52:30.7061000Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7061450Z 2025-10-10T01:52:30.7062353Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:30.7063171Z 2025-10-10T01:52:30.7063694Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:30.7064199Z 2025-10-10T01:52:30.7064575Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:30.7064992Z 2025-10-10T01:52:30.7065832Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:30.7066684Z 2025-10-10T01:52:30.7067132Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] raise RuntimeError( 2025-10-10T01:52:30.7067583Z 2025-10-10T01:52:30.7068555Z (EngineCore_DP0 pid=8195) ERROR 10-10 01:52:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:30.7069726Z (EngineCore_DP0 pid=8195) Process EngineCore_DP0: 2025-10-10T01:52:30.7070439Z (EngineCore_DP0 pid=8195) Traceback (most recent call last): 2025-10-10T01:52:30.7071454Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:30.7072314Z (EngineCore_DP0 pid=8195) self.run() 2025-10-10T01:52:30.7073175Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:30.7074110Z (EngineCore_DP0 pid=8195) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:30.7075229Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:30.7076188Z (EngineCore_DP0 pid=8195) raise e 2025-10-10T01:52:30.7077230Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:30.7078457Z (EngineCore_DP0 pid=8195) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:30.7079320Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7080390Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:30.7081467Z (EngineCore_DP0 pid=8195) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:30.7082601Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:30.7083650Z (EngineCore_DP0 pid=8195) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:30.7084401Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7085503Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:30.7086566Z (EngineCore_DP0 pid=8195) self._init_executor() 2025-10-10T01:52:30.7087711Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:30.7088877Z (EngineCore_DP0 pid=8195) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:30.7090206Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:30.7091536Z (EngineCore_DP0 pid=8195) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:30.7092338Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7093359Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:30.7094399Z (EngineCore_DP0 pid=8195) return func(*args, **kwargs) 2025-10-10T01:52:30.7095085Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7096469Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:30.7097631Z (EngineCore_DP0 pid=8195) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:30.7098372Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7099539Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:30.7100838Z (EngineCore_DP0 pid=8195) module = importlib.import_module(module_name) 2025-10-10T01:52:30.7101686Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7102762Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:30.7103907Z (EngineCore_DP0 pid=8195) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:30.7104780Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7105574Z (EngineCore_DP0 pid=8195) File "", line 1387, in _gcd_import 2025-10-10T01:52:30.7106618Z (EngineCore_DP0 pid=8195) File "", line 1360, in _find_and_load 2025-10-10T01:52:30.7107681Z (EngineCore_DP0 pid=8195) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:30.7108777Z (EngineCore_DP0 pid=8195) File "", line 935, in _load_unlocked 2025-10-10T01:52:30.7109756Z (EngineCore_DP0 pid=8195) File "", line 999, in exec_module 2025-10-10T01:52:30.7110784Z (EngineCore_DP0 pid=8195) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:30.7112089Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:30.7113333Z (EngineCore_DP0 pid=8195) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:30.7114657Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:30.7115966Z (EngineCore_DP0 pid=8195) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:30.7117466Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:30.7118681Z (EngineCore_DP0 pid=8195) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:30.7120117Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:30.7121546Z (EngineCore_DP0 pid=8195) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:30.7122375Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7123582Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:30.7124809Z (EngineCore_DP0 pid=8195) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:30.7125522Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7126888Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:30.7128160Z (EngineCore_DP0 pid=8195) return _is_fa2_supported(device)[0] 2025-10-10T01:52:30.7128896Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7130194Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:30.7131504Z (EngineCore_DP0 pid=8195) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:30.7132384Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7133629Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:30.7134802Z (EngineCore_DP0 pid=8195) prop = get_device_properties(device) 2025-10-10T01:52:30.7135551Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:30.7136744Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:30.7138030Z (EngineCore_DP0 pid=8195) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:30.7138694Z (EngineCore_DP0 pid=8195) ^^^^^^^^^^^^ 2025-10-10T01:52:30.7139603Z (EngineCore_DP0 pid=8195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:30.7140551Z (EngineCore_DP0 pid=8195) raise RuntimeError( 2025-10-10T01:52:30.7141690Z (EngineCore_DP0 pid=8195) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:31.1282222Z FAILED 2025-10-10T01:52:31.1411873Z models/test_initialization.py::test_can_initialize_large_subset[HunYuanDenseV1ForCausalLM] Fork a new process to run a test 8199 2025-10-10T01:52:31.1422775Z Fork a new process to run a test 0 2025-10-10T01:52:31.1424666Z Model is not available online 2025-10-10T01:52:31.4454957Z PASSED 2025-10-10T01:52:31.4584791Z models/test_initialization.py::test_can_initialize_large_subset[MiniMaxForCausalLM] Fork a new process to run a test 8200 2025-10-10T01:52:31.4596007Z Fork a new process to run a test 0 2025-10-10T01:52:31.4879963Z INFO 10-10 01:52:31 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiniMaxForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'MiniMaxAI/MiniMax-Text-01-hf'} 2025-10-10T01:52:31.6492443Z 2025-10-10T01:52:31.6494633Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:31.6494944Z config.json: 3.00kB [00:00, 13.5MB/s] 2025-10-10T01:52:31.7681610Z INFO 10-10 01:52:31 [model.py:551] Resolved architecture: MiniMaxForCausalLM 2025-10-10T01:52:31.7682096Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:52:31.8688119Z 2025-10-10T01:52:31.8711265Z model.safetensors.index.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:31.8711666Z model.safetensors.index.json: 823kB [00:00, 360MB/s] 2025-10-10T01:52:31.8995181Z 2025-10-10T01:52:32.3144085Z Parse safetensors files: 0% 0/413 [00:00= mamba page size. 2025-10-10T01:52:33.5268367Z 2025-10-10T01:52:33.5269784Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:33.5270141Z tokenizer_config.json: 1.39kB [00:00, 12.3MB/s] 2025-10-10T01:52:33.6347513Z 2025-10-10T01:52:33.6550378Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:33.6550677Z vocab.json: 4.71MB [00:00, 232MB/s] 2025-10-10T01:52:33.7050503Z 2025-10-10T01:52:33.7169916Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:52:33.7170224Z merges.txt: 2.41MB [00:00, 201MB/s] 2025-10-10T01:52:33.8110750Z 2025-10-10T01:52:33.8464369Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:33.8464693Z tokenizer.json: 9.72MB [00:00, 276MB/s] 2025-10-10T01:52:34.6629383Z (EngineCore_DP0 pid=8228) INFO 10-10 01:52:34 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:52:34.6709465Z (EngineCore_DP0 pid=8228) INFO 10-10 01:52:34 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='MiniMaxAI/MiniMax-Text-01-hf', speculative_config=None, tokenizer='MiniMaxAI/MiniMax-Text-01-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=10240000, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=MiniMaxAI/MiniMax-Text-01-hf, enable_prefix_caching=False, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:34.7690160Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:34.7690604Z 2025-10-10T01:52:34.7690983Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:34.7691339Z 2025-10-10T01:52:34.7692037Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:34.7692630Z 2025-10-10T01:52:34.7693025Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:34.7693689Z 2025-10-10T01:52:34.7694079Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7694438Z 2025-10-10T01:52:34.7695058Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:34.7695623Z 2025-10-10T01:52:34.7696036Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:34.7696805Z 2025-10-10T01:52:34.7697360Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:34.7697797Z 2025-10-10T01:52:34.7698119Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:34.7698523Z 2025-10-10T01:52:34.7698812Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7699083Z 2025-10-10T01:52:34.7699585Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:34.7700045Z 2025-10-10T01:52:34.7700297Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] self._init_executor() 2025-10-10T01:52:34.7700555Z 2025-10-10T01:52:34.7701087Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:34.7701585Z 2025-10-10T01:52:34.7701903Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:34.7702257Z 2025-10-10T01:52:34.7702907Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:34.7703418Z 2025-10-10T01:52:34.7703768Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:34.7704124Z 2025-10-10T01:52:34.7704419Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7704709Z 2025-10-10T01:52:34.7705218Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:34.7705673Z 2025-10-10T01:52:34.7705948Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:34.7706221Z 2025-10-10T01:52:34.7706476Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7706742Z 2025-10-10T01:52:34.7707255Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:34.7707736Z 2025-10-10T01:52:34.7708023Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:34.7708317Z 2025-10-10T01:52:34.7708583Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7708856Z 2025-10-10T01:52:34.7709449Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:34.7709944Z 2025-10-10T01:52:34.7710253Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:34.7710564Z 2025-10-10T01:52:34.7710844Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7711118Z 2025-10-10T01:52:34.7711564Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:34.7712029Z 2025-10-10T01:52:34.7712376Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:34.7712704Z 2025-10-10T01:52:34.7712991Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7713313Z 2025-10-10T01:52:34.7713663Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:34.7713997Z 2025-10-10T01:52:34.7714347Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:34.7714681Z 2025-10-10T01:52:34.7715053Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:34.7715416Z 2025-10-10T01:52:34.7715753Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:34.7716080Z 2025-10-10T01:52:34.7716442Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:34.7716800Z 2025-10-10T01:52:34.7717217Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:34.7717576Z 2025-10-10T01:52:34.7718071Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:34.7718526Z 2025-10-10T01:52:34.7718897Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:34.7719377Z 2025-10-10T01:52:34.7719898Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:34.7720377Z 2025-10-10T01:52:34.7720753Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:34.7721119Z 2025-10-10T01:52:34.7721668Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:34.7722173Z 2025-10-10T01:52:34.7722462Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:34.7722762Z 2025-10-10T01:52:34.7723388Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:34.7723960Z 2025-10-10T01:52:34.7724383Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:34.7724739Z 2025-10-10T01:52:34.7725013Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7725277Z 2025-10-10T01:52:34.7725851Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:34.7726367Z 2025-10-10T01:52:34.7726659Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:34.7727010Z 2025-10-10T01:52:34.7727281Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7727555Z 2025-10-10T01:52:34.7728152Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:34.7728753Z 2025-10-10T01:52:34.7729033Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:34.7729319Z 2025-10-10T01:52:34.7729574Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7729836Z 2025-10-10T01:52:34.7730415Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:34.7730941Z 2025-10-10T01:52:34.7731260Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:34.7731566Z 2025-10-10T01:52:34.7731840Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7732113Z 2025-10-10T01:52:34.7732681Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:34.7733182Z 2025-10-10T01:52:34.7733476Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:34.7733766Z 2025-10-10T01:52:34.7734024Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7734297Z 2025-10-10T01:52:34.7734812Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:34.7735293Z 2025-10-10T01:52:34.7735599Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:34.7735901Z 2025-10-10T01:52:34.7736131Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:34.7736369Z 2025-10-10T01:52:34.7736851Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:34.7737296Z 2025-10-10T01:52:34.7737542Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] raise RuntimeError( 2025-10-10T01:52:34.7737796Z 2025-10-10T01:52:34.7738357Z (EngineCore_DP0 pid=8228) ERROR 10-10 01:52:34 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:34.7739079Z (EngineCore_DP0 pid=8228) Process EngineCore_DP0: 2025-10-10T01:52:34.7739511Z (EngineCore_DP0 pid=8228) Traceback (most recent call last): 2025-10-10T01:52:34.7740133Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:34.7740664Z (EngineCore_DP0 pid=8228) self.run() 2025-10-10T01:52:34.7741192Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:34.7741754Z (EngineCore_DP0 pid=8228) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:34.7742416Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:34.7743009Z (EngineCore_DP0 pid=8228) raise e 2025-10-10T01:52:34.7743593Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:34.7744291Z (EngineCore_DP0 pid=8228) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:34.7744737Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7745348Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:34.7745982Z (EngineCore_DP0 pid=8228) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:34.7746642Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:34.7747260Z (EngineCore_DP0 pid=8228) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:34.7747711Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7748348Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:34.7748978Z (EngineCore_DP0 pid=8228) self._init_executor() 2025-10-10T01:52:34.7749645Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:34.7750327Z (EngineCore_DP0 pid=8228) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:34.7751045Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:34.7751764Z (EngineCore_DP0 pid=8228) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:34.7752259Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7752897Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:34.7753505Z (EngineCore_DP0 pid=8228) return func(*args, **kwargs) 2025-10-10T01:52:34.7760681Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7761416Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:34.7762059Z (EngineCore_DP0 pid=8228) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:34.7762512Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7763204Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:34.7763951Z (EngineCore_DP0 pid=8228) module = importlib.import_module(module_name) 2025-10-10T01:52:34.7764419Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7765004Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:34.7765641Z (EngineCore_DP0 pid=8228) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:34.7766142Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7766666Z (EngineCore_DP0 pid=8228) File "", line 1387, in _gcd_import 2025-10-10T01:52:34.7767316Z (EngineCore_DP0 pid=8228) File "", line 1360, in _find_and_load 2025-10-10T01:52:34.7767900Z (EngineCore_DP0 pid=8228) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:34.7768540Z (EngineCore_DP0 pid=8228) File "", line 935, in _load_unlocked 2025-10-10T01:52:34.7769120Z (EngineCore_DP0 pid=8228) File "", line 999, in exec_module 2025-10-10T01:52:34.7769730Z (EngineCore_DP0 pid=8228) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:34.7770472Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:34.7771158Z (EngineCore_DP0 pid=8228) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:34.7771922Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:34.7772659Z (EngineCore_DP0 pid=8228) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:34.7773507Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:34.7774192Z (EngineCore_DP0 pid=8228) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:34.7775004Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:34.7775821Z (EngineCore_DP0 pid=8228) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:34.7776308Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7777008Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:34.7777695Z (EngineCore_DP0 pid=8228) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:34.7778140Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7778890Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:34.7779587Z (EngineCore_DP0 pid=8228) return _is_fa2_supported(device)[0] 2025-10-10T01:52:34.7780019Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7780734Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:34.7781449Z (EngineCore_DP0 pid=8228) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:34.7781955Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7782650Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:34.7783288Z (EngineCore_DP0 pid=8228) prop = get_device_properties(device) 2025-10-10T01:52:34.7783713Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:34.7784372Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:34.7785081Z (EngineCore_DP0 pid=8228) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:34.7785471Z (EngineCore_DP0 pid=8228) ^^^^^^^^^^^^ 2025-10-10T01:52:34.7786062Z (EngineCore_DP0 pid=8228) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:34.7786676Z (EngineCore_DP0 pid=8228) raise RuntimeError( 2025-10-10T01:52:34.7787381Z (EngineCore_DP0 pid=8228) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:35.1781600Z FAILED 2025-10-10T01:52:35.1911201Z models/test_initialization.py::test_can_initialize_large_subset[EagleMiniCPMForCausalLM] Fork a new process to run a test 8232 2025-10-10T01:52:35.1923298Z Fork a new process to run a test 0 2025-10-10T01:52:35.1924840Z Model is not available online 2025-10-10T01:52:35.4935982Z PASSED 2025-10-10T01:52:35.5065746Z models/test_initialization.py::test_can_initialize_large_subset[Olmo3ForCausalLM] Fork a new process to run a test 8233 2025-10-10T01:52:35.5077500Z Fork a new process to run a test 0 2025-10-10T01:52:35.5352527Z INFO 10-10 01:52:35 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Olmo3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'shanearora/2025-sep-a-base-model'} 2025-10-10T01:52:35.6448168Z 2025-10-10T01:52:35.6450339Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:35.6450651Z config.json: 1.46kB [00:00, 8.13MB/s] 2025-10-10T01:52:35.7818381Z INFO 10-10 01:52:35 [model.py:551] Resolved architecture: Olmo2ForCausalLM 2025-10-10T01:52:35.7818854Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:52:35.8070334Z INFO 10-10 01:52:35 [model.py:1545] Using max model len 8192 2025-10-10T01:52:35.9793873Z INFO 10-10 01:52:35 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:52:36.0402920Z 2025-10-10T01:52:36.0404908Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:36.0405318Z tokenizer_config.json: 4.34kB [00:00, 31.3MB/s] 2025-10-10T01:52:36.1251903Z 2025-10-10T01:52:36.1371859Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:36.1372155Z vocab.json: 1.61MB [00:00, 135MB/s] 2025-10-10T01:52:36.1809625Z 2025-10-10T01:52:36.1858500Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:52:36.1858870Z merges.txt: 917kB [00:00, 187MB/s] 2025-10-10T01:52:36.2230698Z 2025-10-10T01:52:36.2437102Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:36.2437536Z tokenizer.json: 7.14MB [00:00, 347MB/s] 2025-10-10T01:52:36.3239120Z 2025-10-10T01:52:36.3240097Z special_tokens_map.json: 0% 0.00/125 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:36.8044769Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:36.8045190Z 2025-10-10T01:52:36.8045580Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:36.8045940Z 2025-10-10T01:52:36.8046629Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:36.8047262Z 2025-10-10T01:52:36.8047644Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:36.8048023Z 2025-10-10T01:52:36.8048372Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8048713Z 2025-10-10T01:52:36.8049217Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:36.8049681Z 2025-10-10T01:52:36.8050012Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:36.8050328Z 2025-10-10T01:52:36.8050802Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:36.8051237Z 2025-10-10T01:52:36.8051560Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:36.8051869Z 2025-10-10T01:52:36.8052239Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8052518Z 2025-10-10T01:52:36.8053042Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:36.8053515Z 2025-10-10T01:52:36.8053767Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] self._init_executor() 2025-10-10T01:52:36.8054034Z 2025-10-10T01:52:36.8054568Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:36.8055154Z 2025-10-10T01:52:36.8055652Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:36.8056252Z 2025-10-10T01:52:36.8056935Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:36.8057942Z 2025-10-10T01:52:36.8058580Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:36.8059182Z 2025-10-10T01:52:36.8059526Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8059814Z 2025-10-10T01:52:36.8060432Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:36.8060902Z 2025-10-10T01:52:36.8061185Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:36.8061457Z 2025-10-10T01:52:36.8061718Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8061982Z 2025-10-10T01:52:36.8062569Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:36.8063062Z 2025-10-10T01:52:36.8063352Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:36.8063653Z 2025-10-10T01:52:36.8063947Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8064217Z 2025-10-10T01:52:36.8064741Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:36.8065232Z 2025-10-10T01:52:36.8065539Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:36.8065845Z 2025-10-10T01:52:36.8066116Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8066399Z 2025-10-10T01:52:36.8066847Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:36.8067259Z 2025-10-10T01:52:36.8067601Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:36.8067940Z 2025-10-10T01:52:36.8068239Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8068524Z 2025-10-10T01:52:36.8068919Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:36.8069255Z 2025-10-10T01:52:36.8069607Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:36.8069948Z 2025-10-10T01:52:36.8070318Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:36.8070682Z 2025-10-10T01:52:36.8071019Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:36.8071394Z 2025-10-10T01:52:36.8071753Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:36.8072109Z 2025-10-10T01:52:36.8072481Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:36.8072877Z 2025-10-10T01:52:36.8073387Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:36.8073843Z 2025-10-10T01:52:36.8074197Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:36.8074532Z 2025-10-10T01:52:36.8075055Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:36.8075532Z 2025-10-10T01:52:36.8075908Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:36.8076269Z 2025-10-10T01:52:36.8076857Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:36.8077368Z 2025-10-10T01:52:36.8077658Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:36.8077957Z 2025-10-10T01:52:36.8078592Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:36.8079327Z 2025-10-10T01:52:36.8079698Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:36.8080054Z 2025-10-10T01:52:36.8080313Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8080575Z 2025-10-10T01:52:36.8081144Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:36.8081660Z 2025-10-10T01:52:36.8081963Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:36.8082255Z 2025-10-10T01:52:36.8082529Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8082804Z 2025-10-10T01:52:36.8083397Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:36.8083950Z 2025-10-10T01:52:36.8084278Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:36.8084575Z 2025-10-10T01:52:36.8084839Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8085110Z 2025-10-10T01:52:36.8085683Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:36.8086240Z 2025-10-10T01:52:36.8086571Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:36.8086925Z 2025-10-10T01:52:36.8087208Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8087483Z 2025-10-10T01:52:36.8088022Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:36.8088549Z 2025-10-10T01:52:36.8088836Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:36.8089122Z 2025-10-10T01:52:36.8089383Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8089665Z 2025-10-10T01:52:36.8090190Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:36.8090680Z 2025-10-10T01:52:36.8090989Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:36.8091297Z 2025-10-10T01:52:36.8091522Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:36.8091776Z 2025-10-10T01:52:36.8092881Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:36.8093346Z 2025-10-10T01:52:36.8093609Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] raise RuntimeError( 2025-10-10T01:52:36.8093867Z 2025-10-10T01:52:36.8094442Z (EngineCore_DP0 pid=8241) ERROR 10-10 01:52:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:36.8095140Z (EngineCore_DP0 pid=8241) Process EngineCore_DP0: 2025-10-10T01:52:36.8095530Z (EngineCore_DP0 pid=8241) Traceback (most recent call last): 2025-10-10T01:52:36.8096359Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:36.8096896Z (EngineCore_DP0 pid=8241) self.run() 2025-10-10T01:52:36.8097443Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:36.8098008Z (EngineCore_DP0 pid=8241) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:36.8098678Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:36.8099225Z (EngineCore_DP0 pid=8241) raise e 2025-10-10T01:52:36.8099820Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:36.8100450Z (EngineCore_DP0 pid=8241) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:36.8100984Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8101617Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:36.8102265Z (EngineCore_DP0 pid=8241) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:36.8102925Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:36.8103545Z (EngineCore_DP0 pid=8241) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:36.8104072Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8104716Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:36.8105379Z (EngineCore_DP0 pid=8241) self._init_executor() 2025-10-10T01:52:36.8106040Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:36.8106723Z (EngineCore_DP0 pid=8241) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:36.8107462Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:36.8108167Z (EngineCore_DP0 pid=8241) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:36.8108664Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8109308Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:36.8109897Z (EngineCore_DP0 pid=8241) return func(*args, **kwargs) 2025-10-10T01:52:36.8110358Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8111009Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:36.8111638Z (EngineCore_DP0 pid=8241) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:36.8112063Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8112730Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:36.8113398Z (EngineCore_DP0 pid=8241) module = importlib.import_module(module_name) 2025-10-10T01:52:36.8113844Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8114430Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:36.8115054Z (EngineCore_DP0 pid=8241) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:36.8115549Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8116042Z (EngineCore_DP0 pid=8241) File "", line 1387, in _gcd_import 2025-10-10T01:52:36.8116605Z (EngineCore_DP0 pid=8241) File "", line 1360, in _find_and_load 2025-10-10T01:52:36.8117202Z (EngineCore_DP0 pid=8241) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:36.8117782Z (EngineCore_DP0 pid=8241) File "", line 935, in _load_unlocked 2025-10-10T01:52:36.8118394Z (EngineCore_DP0 pid=8241) File "", line 999, in exec_module 2025-10-10T01:52:36.8119098Z (EngineCore_DP0 pid=8241) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:36.8119839Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:36.8120512Z (EngineCore_DP0 pid=8241) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:36.8121232Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:36.8121997Z (EngineCore_DP0 pid=8241) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:36.8122790Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:36.8123505Z (EngineCore_DP0 pid=8241) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:36.8124286Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:36.8125101Z (EngineCore_DP0 pid=8241) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:36.8125584Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8126288Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:36.8126972Z (EngineCore_DP0 pid=8241) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:36.8127416Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8128201Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:36.8128913Z (EngineCore_DP0 pid=8241) return _is_fa2_supported(device)[0] 2025-10-10T01:52:36.8129331Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8130045Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:36.8130762Z (EngineCore_DP0 pid=8241) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:36.8131212Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8131886Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:36.8132529Z (EngineCore_DP0 pid=8241) prop = get_device_properties(device) 2025-10-10T01:52:36.8132964Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:36.8133622Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:36.8134277Z (EngineCore_DP0 pid=8241) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:36.8134686Z (EngineCore_DP0 pid=8241) ^^^^^^^^^^^^ 2025-10-10T01:52:36.8135267Z (EngineCore_DP0 pid=8241) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:36.8135831Z (EngineCore_DP0 pid=8241) raise RuntimeError( 2025-10-10T01:52:36.8136553Z (EngineCore_DP0 pid=8241) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:37.2103456Z FAILED 2025-10-10T01:52:37.2232494Z models/test_initialization.py::test_can_initialize_large_subset[NemotronH_Nano_VL_V2] Fork a new process to run a test 8245 2025-10-10T01:52:37.2244316Z Fork a new process to run a test 0 2025-10-10T01:52:37.2245760Z Model is not available online 2025-10-10T01:52:37.5276217Z PASSED 2025-10-10T01:52:37.5406078Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2MoeForCausalLM] Fork a new process to run a test 8246 2025-10-10T01:52:37.5417445Z Fork a new process to run a test 0 2025-10-10T01:52:37.5688443Z INFO 10-10 01:52:37 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2MoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen1.5-MoE-A2.7B-Chat'} 2025-10-10T01:52:37.6722777Z 2025-10-10T01:52:37.6723844Z config.json: 0% 0.00/920 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:45.4170212Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:45.4170890Z 2025-10-10T01:52:45.4171569Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:45.4171949Z 2025-10-10T01:52:45.4172632Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:45.4173246Z 2025-10-10T01:52:45.4173632Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:45.4174016Z 2025-10-10T01:52:45.4174357Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4174703Z 2025-10-10T01:52:45.4175319Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:45.4175873Z 2025-10-10T01:52:45.4176288Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:45.4176970Z 2025-10-10T01:52:45.4177617Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:45.4178165Z 2025-10-10T01:52:45.4178557Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:45.4178933Z 2025-10-10T01:52:45.4179281Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4179621Z 2025-10-10T01:52:45.4180247Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:45.4181110Z 2025-10-10T01:52:45.4181744Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] self._init_executor() 2025-10-10T01:52:45.4182397Z 2025-10-10T01:52:45.4183526Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:45.4184467Z 2025-10-10T01:52:45.4185104Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:45.4185464Z 2025-10-10T01:52:45.4186028Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:45.4186546Z 2025-10-10T01:52:45.4186900Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:45.4187353Z 2025-10-10T01:52:45.4187683Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4187971Z 2025-10-10T01:52:45.4188476Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:45.4188928Z 2025-10-10T01:52:45.4189369Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:45.4189746Z 2025-10-10T01:52:45.4190015Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4190280Z 2025-10-10T01:52:45.4190795Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:45.4191320Z 2025-10-10T01:52:45.4191610Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:45.4192033Z 2025-10-10T01:52:45.4192339Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4192608Z 2025-10-10T01:52:45.4193129Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:45.4193610Z 2025-10-10T01:52:45.4193909Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:45.4194211Z 2025-10-10T01:52:45.4194480Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4194757Z 2025-10-10T01:52:45.4195260Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:45.4195673Z 2025-10-10T01:52:45.4196018Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:45.4196559Z 2025-10-10T01:52:45.4196861Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4197142Z 2025-10-10T01:52:45.4197490Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:45.4197826Z 2025-10-10T01:52:45.4198183Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:45.4198521Z 2025-10-10T01:52:45.4198908Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:45.4199381Z 2025-10-10T01:52:45.4199732Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:45.4200314Z 2025-10-10T01:52:45.4200839Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:45.4201205Z 2025-10-10T01:52:45.4201597Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:45.4201967Z 2025-10-10T01:52:45.4202476Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:45.4203052Z 2025-10-10T01:52:45.4203420Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:45.4203757Z 2025-10-10T01:52:45.4204281Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:45.4204782Z 2025-10-10T01:52:45.4205156Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:45.4205569Z 2025-10-10T01:52:45.4206128Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:45.4206627Z 2025-10-10T01:52:45.4206940Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:45.4207306Z 2025-10-10T01:52:45.4207947Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:45.4208515Z 2025-10-10T01:52:45.4208884Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:45.4209234Z 2025-10-10T01:52:45.4209485Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4209749Z 2025-10-10T01:52:45.4210305Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:45.4210823Z 2025-10-10T01:52:45.4211114Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:45.4211472Z 2025-10-10T01:52:45.4211744Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4212014Z 2025-10-10T01:52:45.4212609Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:45.4213153Z 2025-10-10T01:52:45.4213437Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:45.4213713Z 2025-10-10T01:52:45.4213972Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4214232Z 2025-10-10T01:52:45.4214813Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:45.4215337Z 2025-10-10T01:52:45.4215651Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:45.4215960Z 2025-10-10T01:52:45.4216226Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4216498Z 2025-10-10T01:52:45.4217012Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:45.4217503Z 2025-10-10T01:52:45.4217788Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:45.4218069Z 2025-10-10T01:52:45.4218375Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4218645Z 2025-10-10T01:52:45.4219164Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:45.4219636Z 2025-10-10T01:52:45.4219947Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:45.4220244Z 2025-10-10T01:52:45.4220511Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:45.4220758Z 2025-10-10T01:52:45.4221237Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:45.4221687Z 2025-10-10T01:52:45.4221977Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] raise RuntimeError( 2025-10-10T01:52:45.4222239Z 2025-10-10T01:52:45.4222801Z (EngineCore_DP0 pid=8304) ERROR 10-10 01:52:45 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:45.4223506Z (EngineCore_DP0 pid=8304) Process EngineCore_DP0: 2025-10-10T01:52:45.4223902Z (EngineCore_DP0 pid=8304) Traceback (most recent call last): 2025-10-10T01:52:45.4224503Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:45.4225008Z (EngineCore_DP0 pid=8304) self.run() 2025-10-10T01:52:45.4225553Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:45.4226127Z (EngineCore_DP0 pid=8304) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:45.4226835Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:45.4227383Z (EngineCore_DP0 pid=8304) raise e 2025-10-10T01:52:45.4227973Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:45.4228604Z (EngineCore_DP0 pid=8304) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:45.4229049Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4229664Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:45.4230301Z (EngineCore_DP0 pid=8304) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:45.4230952Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:45.4231567Z (EngineCore_DP0 pid=8304) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:45.4232033Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4232680Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:45.4233261Z (EngineCore_DP0 pid=8304) self._init_executor() 2025-10-10T01:52:45.4233913Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:45.4234664Z (EngineCore_DP0 pid=8304) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:45.4235396Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:45.4236109Z (EngineCore_DP0 pid=8304) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:45.4236605Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4237231Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:45.4237856Z (EngineCore_DP0 pid=8304) return func(*args, **kwargs) 2025-10-10T01:52:45.4238238Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4238870Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:45.4239629Z (EngineCore_DP0 pid=8304) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:45.4240051Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4240712Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:45.4241357Z (EngineCore_DP0 pid=8304) module = importlib.import_module(module_name) 2025-10-10T01:52:45.4241793Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4242367Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:45.4242985Z (EngineCore_DP0 pid=8304) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:45.4243463Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4244001Z (EngineCore_DP0 pid=8304) File "", line 1387, in _gcd_import 2025-10-10T01:52:45.4244554Z (EngineCore_DP0 pid=8304) File "", line 1360, in _find_and_load 2025-10-10T01:52:45.4245134Z (EngineCore_DP0 pid=8304) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:45.4245707Z (EngineCore_DP0 pid=8304) File "", line 935, in _load_unlocked 2025-10-10T01:52:45.4246286Z (EngineCore_DP0 pid=8304) File "", line 999, in exec_module 2025-10-10T01:52:45.4246876Z (EngineCore_DP0 pid=8304) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:45.4247622Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:45.4248305Z (EngineCore_DP0 pid=8304) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:45.4249026Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:45.4249748Z (EngineCore_DP0 pid=8304) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:45.4250523Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:45.4251192Z (EngineCore_DP0 pid=8304) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:45.4252023Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:45.4252841Z (EngineCore_DP0 pid=8304) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:45.4253318Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4254005Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:45.4254680Z (EngineCore_DP0 pid=8304) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:45.4255114Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4255896Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:45.4256586Z (EngineCore_DP0 pid=8304) return _is_fa2_supported(device)[0] 2025-10-10T01:52:45.4257060Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4257762Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:45.4258470Z (EngineCore_DP0 pid=8304) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:45.4258915Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4259576Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:45.4260220Z (EngineCore_DP0 pid=8304) prop = get_device_properties(device) 2025-10-10T01:52:45.4260633Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:45.4261284Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:45.4261982Z (EngineCore_DP0 pid=8304) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:45.4262378Z (EngineCore_DP0 pid=8304) ^^^^^^^^^^^^ 2025-10-10T01:52:45.4262951Z (EngineCore_DP0 pid=8304) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:45.4263506Z (EngineCore_DP0 pid=8304) raise RuntimeError( 2025-10-10T01:52:45.4264178Z (EngineCore_DP0 pid=8304) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:45.8248326Z FAILED 2025-10-10T01:52:45.8377710Z models/test_initialization.py::test_can_initialize_large_subset[GraniteSpeechForConditionalGeneration] Fork a new process to run a test 8308 2025-10-10T01:52:45.8388650Z Fork a new process to run a test 0 2025-10-10T01:52:45.8668964Z INFO 10-10 01:52:45 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GraniteSpeechForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ibm-granite/granite-speech-3.3-2b'} 2025-10-10T01:52:45.9469933Z 2025-10-10T01:52:45.9472458Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:45.9472758Z config.json: 2.41kB [00:00, 10.4MB/s] 2025-10-10T01:52:46.0758871Z 2025-10-10T01:52:46.0760507Z preprocessor_config.json: 0% 0.00/2.00 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:53.8016853Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:53.8017432Z 2025-10-10T01:52:53.8017862Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:53.8018267Z 2025-10-10T01:52:53.8018948Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:53.8019541Z 2025-10-10T01:52:53.8019934Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:53.8020483Z 2025-10-10T01:52:53.8020829Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8021167Z 2025-10-10T01:52:53.8021781Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:53.8022464Z 2025-10-10T01:52:53.8023025Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:53.8023600Z 2025-10-10T01:52:53.8024335Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:53.8025008Z 2025-10-10T01:52:53.8025412Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:53.8025781Z 2025-10-10T01:52:53.8026080Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8026359Z 2025-10-10T01:52:53.8026868Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:53.8027502Z 2025-10-10T01:52:53.8028001Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] self._init_executor() 2025-10-10T01:52:53.8028414Z 2025-10-10T01:52:53.8029022Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:53.8029697Z 2025-10-10T01:52:53.8030044Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:53.8030368Z 2025-10-10T01:52:53.8030984Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:53.8031752Z 2025-10-10T01:52:53.8032307Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:53.8032671Z 2025-10-10T01:52:53.8032983Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8033268Z 2025-10-10T01:52:53.8033866Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:53.8034589Z 2025-10-10T01:52:53.8035124Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:53.8035613Z 2025-10-10T01:52:53.8036064Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8036531Z 2025-10-10T01:52:53.8037486Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:53.8038263Z 2025-10-10T01:52:53.8038759Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:53.8039345Z 2025-10-10T01:52:53.8039772Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8040202Z 2025-10-10T01:52:53.8041030Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:53.8041867Z 2025-10-10T01:52:53.8042343Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:53.8042818Z 2025-10-10T01:52:53.8043307Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8043752Z 2025-10-10T01:52:53.8044459Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:53.8045125Z 2025-10-10T01:52:53.8045702Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:53.8046262Z 2025-10-10T01:52:53.8046756Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8047228Z 2025-10-10T01:52:53.8047805Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:53.8048383Z 2025-10-10T01:52:53.8049021Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:53.8049691Z 2025-10-10T01:52:53.8050386Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:53.8051061Z 2025-10-10T01:52:53.8051653Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:53.8052227Z 2025-10-10T01:52:53.8052842Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:53.8053437Z 2025-10-10T01:52:53.8054087Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:53.8054704Z 2025-10-10T01:52:53.8055579Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:53.8056396Z 2025-10-10T01:52:53.8057010Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:53.8057600Z 2025-10-10T01:52:53.8058518Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:53.8059393Z 2025-10-10T01:52:53.8060092Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:53.8060781Z 2025-10-10T01:52:53.8061825Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:53.8062745Z 2025-10-10T01:52:53.8063254Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:53.8063818Z 2025-10-10T01:52:53.8065123Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:53.8066245Z 2025-10-10T01:52:53.8066878Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:53.8067540Z 2025-10-10T01:52:53.8067970Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8068404Z 2025-10-10T01:52:53.8069363Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:53.8070301Z 2025-10-10T01:52:53.8070817Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:53.8071374Z 2025-10-10T01:52:53.8071876Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8072375Z 2025-10-10T01:52:53.8073491Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:53.8074723Z 2025-10-10T01:52:53.8075331Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:53.8075822Z 2025-10-10T01:52:53.8076256Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8076690Z 2025-10-10T01:52:53.8077713Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:53.8078616Z 2025-10-10T01:52:53.8079220Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:53.8079749Z 2025-10-10T01:52:53.8080225Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8080693Z 2025-10-10T01:52:53.8081610Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:53.8082445Z 2025-10-10T01:52:53.8082922Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:53.8083429Z 2025-10-10T01:52:53.8083867Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8084333Z 2025-10-10T01:52:53.8085259Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:53.8086181Z 2025-10-10T01:52:53.8086749Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:53.8087284Z 2025-10-10T01:52:53.8087666Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:53.8088072Z 2025-10-10T01:52:53.8088959Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:53.8089745Z 2025-10-10T01:52:53.8090171Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] raise RuntimeError( 2025-10-10T01:52:53.8090601Z 2025-10-10T01:52:53.8091574Z (EngineCore_DP0 pid=8366) ERROR 10-10 01:52:53 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:53.8092736Z (EngineCore_DP0 pid=8366) Process EngineCore_DP0: 2025-10-10T01:52:53.8093449Z (EngineCore_DP0 pid=8366) Traceback (most recent call last): 2025-10-10T01:52:53.8094474Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:53.8095328Z (EngineCore_DP0 pid=8366) self.run() 2025-10-10T01:52:53.8096509Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:53.8097557Z (EngineCore_DP0 pid=8366) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:53.8098788Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:53.8099738Z (EngineCore_DP0 pid=8366) raise e 2025-10-10T01:52:53.8100741Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:53.8101862Z (EngineCore_DP0 pid=8366) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:53.8102594Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8103643Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:53.8104828Z (EngineCore_DP0 pid=8366) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:53.8105985Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:53.8107050Z (EngineCore_DP0 pid=8366) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:53.8107832Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8108967Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:53.8110053Z (EngineCore_DP0 pid=8366) self._init_executor() 2025-10-10T01:52:53.8111281Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:53.8112403Z (EngineCore_DP0 pid=8366) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:53.8113603Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:53.8114878Z (EngineCore_DP0 pid=8366) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:53.8115746Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8116852Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:53.8117851Z (EngineCore_DP0 pid=8366) return func(*args, **kwargs) 2025-10-10T01:52:53.8118538Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8119792Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:53.8120889Z (EngineCore_DP0 pid=8366) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:53.8121634Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8122867Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:53.8124170Z (EngineCore_DP0 pid=8366) module = importlib.import_module(module_name) 2025-10-10T01:52:53.8124979Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8126040Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:53.8127277Z (EngineCore_DP0 pid=8366) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:53.8128185Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8129085Z (EngineCore_DP0 pid=8366) File "", line 1387, in _gcd_import 2025-10-10T01:52:53.8130073Z (EngineCore_DP0 pid=8366) File "", line 1360, in _find_and_load 2025-10-10T01:52:53.8131121Z (EngineCore_DP0 pid=8366) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:53.8132155Z (EngineCore_DP0 pid=8366) File "", line 935, in _load_unlocked 2025-10-10T01:52:53.8133154Z (EngineCore_DP0 pid=8366) File "", line 999, in exec_module 2025-10-10T01:52:53.8134212Z (EngineCore_DP0 pid=8366) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:53.8135549Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:53.8136673Z (EngineCore_DP0 pid=8366) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:53.8137849Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:53.8139024Z (EngineCore_DP0 pid=8366) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:53.8140431Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:53.8141667Z (EngineCore_DP0 pid=8366) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:53.8143134Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:53.8144644Z (EngineCore_DP0 pid=8366) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:53.8145443Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8146587Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:53.8147732Z (EngineCore_DP0 pid=8366) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:53.8148459Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8149768Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:53.8150915Z (EngineCore_DP0 pid=8366) return _is_fa2_supported(device)[0] 2025-10-10T01:52:53.8151604Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8152772Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:53.8153944Z (EngineCore_DP0 pid=8366) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:53.8154730Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8155865Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:53.8157058Z (EngineCore_DP0 pid=8366) prop = get_device_properties(device) 2025-10-10T01:52:53.8157811Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:53.8158948Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:53.8160096Z (EngineCore_DP0 pid=8366) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:53.8160724Z (EngineCore_DP0 pid=8366) ^^^^^^^^^^^^ 2025-10-10T01:52:53.8161675Z (EngineCore_DP0 pid=8366) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:53.8162597Z (EngineCore_DP0 pid=8366) raise RuntimeError( 2025-10-10T01:52:53.8163697Z (EngineCore_DP0 pid=8366) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:54.2098701Z FAILED 2025-10-10T01:52:54.2229694Z models/test_initialization.py::test_can_initialize_large_subset[AquilaForCausalLM] Fork a new process to run a test 8370 2025-10-10T01:52:54.2242133Z Fork a new process to run a test 0 2025-10-10T01:52:54.2520300Z INFO 10-10 01:52:54 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='AquilaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/AquilaChat2-7B'} 2025-10-10T01:52:54.3887376Z 2025-10-10T01:52:54.3888247Z config.json: 0% 0.00/678 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:52:55.6681502Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] EngineCore failed to start. 2025-10-10T01:52:55.6682060Z 2025-10-10T01:52:55.6682582Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] Traceback (most recent call last): 2025-10-10T01:52:55.6683123Z 2025-10-10T01:52:55.6684084Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:55.6684965Z 2025-10-10T01:52:55.6685930Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:55.6686644Z 2025-10-10T01:52:55.6687307Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6687907Z 2025-10-10T01:52:55.6688872Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:55.6689751Z 2025-10-10T01:52:55.6690455Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:55.6691280Z 2025-10-10T01:52:55.6692299Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:55.6693175Z 2025-10-10T01:52:55.6694023Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:55.6694646Z 2025-10-10T01:52:55.6695244Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6695795Z 2025-10-10T01:52:55.6697049Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:55.6697963Z 2025-10-10T01:52:55.6698451Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] self._init_executor() 2025-10-10T01:52:55.6698949Z 2025-10-10T01:52:55.6699946Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:55.6700758Z 2025-10-10T01:52:55.6701329Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:55.6702108Z 2025-10-10T01:52:55.6703067Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:55.6703666Z 2025-10-10T01:52:55.6704067Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:55.6704407Z 2025-10-10T01:52:55.6704703Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6704981Z 2025-10-10T01:52:55.6705487Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:55.6705970Z 2025-10-10T01:52:55.6706239Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:52:55.6706513Z 2025-10-10T01:52:55.6706759Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6707018Z 2025-10-10T01:52:55.6707534Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:55.6708010Z 2025-10-10T01:52:55.6708295Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:55.6708580Z 2025-10-10T01:52:55.6708862Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6709129Z 2025-10-10T01:52:55.6709778Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:55.6710278Z 2025-10-10T01:52:55.6710603Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:52:55.6710908Z 2025-10-10T01:52:55.6711179Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6711454Z 2025-10-10T01:52:55.6711889Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:55.6712401Z 2025-10-10T01:52:55.6712749Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:55.6713082Z 2025-10-10T01:52:55.6713463Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6713751Z 2025-10-10T01:52:55.6714092Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:52:55.6714432Z 2025-10-10T01:52:55.6714794Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:52:55.6715149Z 2025-10-10T01:52:55.6715600Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:55.6716037Z 2025-10-10T01:52:55.6716446Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:52:55.6716837Z 2025-10-10T01:52:55.6717200Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:52:55.6717614Z 2025-10-10T01:52:55.6717993Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:55.6718358Z 2025-10-10T01:52:55.6718866Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:55.6719478Z 2025-10-10T01:52:55.6719829Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:55.6720180Z 2025-10-10T01:52:55.6720702Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:55.6721178Z 2025-10-10T01:52:55.6721555Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:55.6721918Z 2025-10-10T01:52:55.6722475Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:55.6722989Z 2025-10-10T01:52:55.6723285Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:52:55.6723583Z 2025-10-10T01:52:55.6724218Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:55.6724793Z 2025-10-10T01:52:55.6725215Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:55.6725584Z 2025-10-10T01:52:55.6725848Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6726125Z 2025-10-10T01:52:55.6726686Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:55.6727206Z 2025-10-10T01:52:55.6727499Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:52:55.6727877Z 2025-10-10T01:52:55.6728148Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6728420Z 2025-10-10T01:52:55.6729031Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:55.6729635Z 2025-10-10T01:52:55.6729924Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:52:55.6730206Z 2025-10-10T01:52:55.6730468Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6730735Z 2025-10-10T01:52:55.6731318Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:55.6731855Z 2025-10-10T01:52:55.6732168Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:55.6732494Z 2025-10-10T01:52:55.6732769Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6733089Z 2025-10-10T01:52:55.6733634Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:55.6734147Z 2025-10-10T01:52:55.6734443Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:52:55.6734732Z 2025-10-10T01:52:55.6735011Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6735316Z 2025-10-10T01:52:55.6735944Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:55.6736518Z 2025-10-10T01:52:55.6736904Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:52:55.6737265Z 2025-10-10T01:52:55.6737505Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:52:55.6737753Z 2025-10-10T01:52:55.6738232Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:55.6738684Z 2025-10-10T01:52:55.6738923Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] raise RuntimeError( 2025-10-10T01:52:55.6739195Z 2025-10-10T01:52:55.6739761Z (EngineCore_DP0 pid=8378) ERROR 10-10 01:52:55 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:55.6740504Z (EngineCore_DP0 pid=8378) Process EngineCore_DP0: 2025-10-10T01:52:55.6740923Z (EngineCore_DP0 pid=8378) Traceback (most recent call last): 2025-10-10T01:52:55.6741554Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:52:55.6742063Z (EngineCore_DP0 pid=8378) self.run() 2025-10-10T01:52:55.6742582Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:52:55.6743140Z (EngineCore_DP0 pid=8378) self._target(*self._args, **self._kwargs) 2025-10-10T01:52:55.6743864Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:52:55.6744413Z (EngineCore_DP0 pid=8378) raise e 2025-10-10T01:52:55.6745002Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:52:55.6745796Z (EngineCore_DP0 pid=8378) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:52:55.6746331Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6747025Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:52:55.6747663Z (EngineCore_DP0 pid=8378) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:52:55.6748324Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:52:55.6748929Z (EngineCore_DP0 pid=8378) self.model_executor = executor_class(vllm_config) 2025-10-10T01:52:55.6749382Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6750081Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:52:55.6750685Z (EngineCore_DP0 pid=8378) self._init_executor() 2025-10-10T01:52:55.6751345Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:52:55.6752025Z (EngineCore_DP0 pid=8378) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:52:55.6752749Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:52:55.6753460Z (EngineCore_DP0 pid=8378) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:52:55.6753953Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6754599Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:52:55.6755177Z (EngineCore_DP0 pid=8378) return func(*args, **kwargs) 2025-10-10T01:52:55.6755571Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6756220Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:52:55.6756846Z (EngineCore_DP0 pid=8378) worker_class = resolve_obj_by_qualname( 2025-10-10T01:52:55.6757264Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6757922Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:52:55.6758618Z (EngineCore_DP0 pid=8378) module = importlib.import_module(module_name) 2025-10-10T01:52:55.6759148Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6759739Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:52:55.6760364Z (EngineCore_DP0 pid=8378) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:52:55.6760855Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6761413Z (EngineCore_DP0 pid=8378) File "", line 1387, in _gcd_import 2025-10-10T01:52:55.6761971Z (EngineCore_DP0 pid=8378) File "", line 1360, in _find_and_load 2025-10-10T01:52:55.6762561Z (EngineCore_DP0 pid=8378) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:52:55.6763227Z (EngineCore_DP0 pid=8378) File "", line 935, in _load_unlocked 2025-10-10T01:52:55.6763802Z (EngineCore_DP0 pid=8378) File "", line 999, in exec_module 2025-10-10T01:52:55.6764395Z (EngineCore_DP0 pid=8378) File "", line 488, in _call_with_frames_removed 2025-10-10T01:52:55.6765124Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:52:55.6765829Z (EngineCore_DP0 pid=8378) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:52:55.6766551Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:52:55.6767365Z (EngineCore_DP0 pid=8378) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:52:55.6768187Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:52:55.6768861Z (EngineCore_DP0 pid=8378) class FlashAttentionMetadataBuilder( 2025-10-10T01:52:55.6769665Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:52:55.6777471Z (EngineCore_DP0 pid=8378) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:52:55.6778007Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6778755Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:52:55.6779492Z (EngineCore_DP0 pid=8378) if not is_fa_version_supported(fa_version): 2025-10-10T01:52:55.6779950Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6780720Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:52:55.6781436Z (EngineCore_DP0 pid=8378) return _is_fa2_supported(device)[0] 2025-10-10T01:52:55.6781861Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6782580Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:52:55.6783311Z (EngineCore_DP0 pid=8378) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:52:55.6783867Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6784581Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:52:55.6785283Z (EngineCore_DP0 pid=8378) prop = get_device_properties(device) 2025-10-10T01:52:55.6785785Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:52:55.6786591Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:52:55.6787471Z (EngineCore_DP0 pid=8378) _lazy_init() # will define _get_device_properties 2025-10-10T01:52:55.6787876Z (EngineCore_DP0 pid=8378) ^^^^^^^^^^^^ 2025-10-10T01:52:55.6788468Z (EngineCore_DP0 pid=8378) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:52:55.6789083Z (EngineCore_DP0 pid=8378) raise RuntimeError( 2025-10-10T01:52:55.6789770Z (EngineCore_DP0 pid=8378) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:52:56.0799408Z FAILED 2025-10-10T01:52:56.0929623Z models/test_initialization.py::test_can_initialize_large_subset[Glm4MoeForCausalLM] Fork a new process to run a test 8382 2025-10-10T01:52:56.0941813Z Fork a new process to run a test 0 2025-10-10T01:52:56.1221194Z INFO 10-10 01:52:56 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Glm4MoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'zai-org/GLM-4.5'} 2025-10-10T01:52:56.2140194Z 2025-10-10T01:52:56.2142106Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:52:56.2142618Z config.json: 1.01kB [00:00, 5.08MB/s] 2025-10-10T01:53:02.8524018Z INFO 10-10 01:53:02 [model.py:551] Resolved architecture: Glm4MoeForCausalLM 2025-10-10T01:53:02.8524541Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:53:02.8769685Z INFO 10-10 01:53:02 [model.py:1545] Using max model len 131072 2025-10-10T01:53:03.0582486Z INFO 10-10 01:53:03 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:53:03.0966747Z 2025-10-10T01:53:03.0968843Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:03.0969266Z tokenizer_config.json: 7.31kB [00:00, 43.3MB/s] 2025-10-10T01:53:03.1925035Z 2025-10-10T01:53:03.6574820Z tokenizer.json: 0% 0.00/20.0M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:04.6106556Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:04.6106986Z 2025-10-10T01:53:04.6107381Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:04.6107737Z 2025-10-10T01:53:04.6108713Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:04.6109322Z 2025-10-10T01:53:04.6109729Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:04.6110108Z 2025-10-10T01:53:04.6110456Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6110813Z 2025-10-10T01:53:04.6111618Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:04.6112202Z 2025-10-10T01:53:04.6112625Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:04.6113035Z 2025-10-10T01:53:04.6113634Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:04.6114148Z 2025-10-10T01:53:04.6114474Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:04.6114816Z 2025-10-10T01:53:04.6115330Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6115726Z 2025-10-10T01:53:04.6116707Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:04.6117621Z 2025-10-10T01:53:04.6118179Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] self._init_executor() 2025-10-10T01:53:04.6118695Z 2025-10-10T01:53:04.6119794Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:04.6120462Z 2025-10-10T01:53:04.6120833Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:04.6121171Z 2025-10-10T01:53:04.6121735Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:04.6122372Z 2025-10-10T01:53:04.6122731Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:04.6123063Z 2025-10-10T01:53:04.6123363Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6123737Z 2025-10-10T01:53:04.6124249Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:04.6124710Z 2025-10-10T01:53:04.6124978Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:04.6125251Z 2025-10-10T01:53:04.6125501Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6125763Z 2025-10-10T01:53:04.6126277Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:04.6126756Z 2025-10-10T01:53:04.6127043Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:04.6127339Z 2025-10-10T01:53:04.6127658Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6127928Z 2025-10-10T01:53:04.6128465Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:04.6128950Z 2025-10-10T01:53:04.6129259Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:04.6129558Z 2025-10-10T01:53:04.6129838Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6130108Z 2025-10-10T01:53:04.6130552Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:04.6130974Z 2025-10-10T01:53:04.6131316Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:04.6131655Z 2025-10-10T01:53:04.6131941Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6132227Z 2025-10-10T01:53:04.6132564Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:04.6132894Z 2025-10-10T01:53:04.6133247Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:04.6133591Z 2025-10-10T01:53:04.6133982Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:04.6134397Z 2025-10-10T01:53:04.6134756Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:04.6135088Z 2025-10-10T01:53:04.6135451Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:04.6135808Z 2025-10-10T01:53:04.6136178Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:04.6136591Z 2025-10-10T01:53:04.6137089Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:04.6137555Z 2025-10-10T01:53:04.6137912Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:04.6138308Z 2025-10-10T01:53:04.6138837Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:04.6139325Z 2025-10-10T01:53:04.6139696Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:04.6140055Z 2025-10-10T01:53:04.6140615Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:04.6141121Z 2025-10-10T01:53:04.6141418Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:04.6141709Z 2025-10-10T01:53:04.6142389Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:04.6142964Z 2025-10-10T01:53:04.6143332Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:04.6143690Z 2025-10-10T01:53:04.6143952Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6144219Z 2025-10-10T01:53:04.6144797Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:04.6145325Z 2025-10-10T01:53:04.6145634Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:04.6145949Z 2025-10-10T01:53:04.6146225Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6146498Z 2025-10-10T01:53:04.6147135Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:04.6147689Z 2025-10-10T01:53:04.6147978Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:04.6148264Z 2025-10-10T01:53:04.6148536Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6148803Z 2025-10-10T01:53:04.6149436Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:04.6149967Z 2025-10-10T01:53:04.6150300Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:04.6150620Z 2025-10-10T01:53:04.6150892Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6151174Z 2025-10-10T01:53:04.6151697Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:04.6152286Z 2025-10-10T01:53:04.6152577Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:04.6152869Z 2025-10-10T01:53:04.6153131Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6153438Z 2025-10-10T01:53:04.6153988Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:04.6154470Z 2025-10-10T01:53:04.6154780Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:04.6155083Z 2025-10-10T01:53:04.6155314Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:04.6155555Z 2025-10-10T01:53:04.6156035Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:04.6156492Z 2025-10-10T01:53:04.6156738Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] raise RuntimeError( 2025-10-10T01:53:04.6157000Z 2025-10-10T01:53:04.6157628Z (EngineCore_DP0 pid=8462) ERROR 10-10 01:53:04 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:04.6158325Z (EngineCore_DP0 pid=8462) Process EngineCore_DP0: 2025-10-10T01:53:04.6158719Z (EngineCore_DP0 pid=8462) Traceback (most recent call last): 2025-10-10T01:53:04.6159451Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:04.6159958Z (EngineCore_DP0 pid=8462) self.run() 2025-10-10T01:53:04.6160484Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:04.6161045Z (EngineCore_DP0 pid=8462) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:04.6161720Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:04.6162274Z (EngineCore_DP0 pid=8462) raise e 2025-10-10T01:53:04.6162857Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:04.6163505Z (EngineCore_DP0 pid=8462) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:04.6163953Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6164576Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:04.6165219Z (EngineCore_DP0 pid=8462) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:04.6165933Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:04.6166567Z (EngineCore_DP0 pid=8462) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:04.6167019Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6167670Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:04.6168254Z (EngineCore_DP0 pid=8462) self._init_executor() 2025-10-10T01:53:04.6168916Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:04.6169675Z (EngineCore_DP0 pid=8462) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:04.6170402Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:04.6171174Z (EngineCore_DP0 pid=8462) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:04.6171671Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6172309Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:04.6172889Z (EngineCore_DP0 pid=8462) return func(*args, **kwargs) 2025-10-10T01:53:04.6173274Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6173914Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:04.6174547Z (EngineCore_DP0 pid=8462) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:04.6174977Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6175696Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:04.6176359Z (EngineCore_DP0 pid=8462) module = importlib.import_module(module_name) 2025-10-10T01:53:04.6176802Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6177382Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:04.6178043Z (EngineCore_DP0 pid=8462) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:04.6178543Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6179046Z (EngineCore_DP0 pid=8462) File "", line 1387, in _gcd_import 2025-10-10T01:53:04.6179609Z (EngineCore_DP0 pid=8462) File "", line 1360, in _find_and_load 2025-10-10T01:53:04.6180198Z (EngineCore_DP0 pid=8462) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:04.6180773Z (EngineCore_DP0 pid=8462) File "", line 935, in _load_unlocked 2025-10-10T01:53:04.6181346Z (EngineCore_DP0 pid=8462) File "", line 999, in exec_module 2025-10-10T01:53:04.6181981Z (EngineCore_DP0 pid=8462) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:04.6182720Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:04.6183452Z (EngineCore_DP0 pid=8462) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:04.6184181Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:04.6184911Z (EngineCore_DP0 pid=8462) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:04.6185686Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:04.6186356Z (EngineCore_DP0 pid=8462) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:04.6187203Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:04.6188015Z (EngineCore_DP0 pid=8462) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:04.6188540Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6189235Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:04.6189914Z (EngineCore_DP0 pid=8462) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:04.6190357Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6191103Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:04.6191806Z (EngineCore_DP0 pid=8462) return _is_fa2_supported(device)[0] 2025-10-10T01:53:04.6192229Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6192991Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:04.6193716Z (EngineCore_DP0 pid=8462) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:04.6194172Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6194841Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:04.6195475Z (EngineCore_DP0 pid=8462) prop = get_device_properties(device) 2025-10-10T01:53:04.6195889Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:04.6196783Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:04.6197454Z (EngineCore_DP0 pid=8462) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:04.6197857Z (EngineCore_DP0 pid=8462) ^^^^^^^^^^^^ 2025-10-10T01:53:04.6198438Z (EngineCore_DP0 pid=8462) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:04.6199065Z (EngineCore_DP0 pid=8462) raise RuntimeError( 2025-10-10T01:53:04.6199764Z (EngineCore_DP0 pid=8462) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:05.0298918Z FAILED 2025-10-10T01:53:05.0429520Z models/test_initialization.py::test_can_initialize_large_subset[PhiMoEForCausalLM] Fork a new process to run a test 8466 2025-10-10T01:53:05.0441197Z Fork a new process to run a test 0 2025-10-10T01:53:05.0718930Z INFO 10-10 01:53:05 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='PhiMoEForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'microsoft/Phi-3.5-MoE-instruct'} 2025-10-10T01:53:05.2162117Z 2025-10-10T01:53:05.2166956Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:05.2167258Z config.json: 4.53kB [00:00, 8.38MB/s] 2025-10-10T01:53:05.2869158Z 2025-10-10T01:53:05.2872377Z configuration_phimoe.py: 0.00B [00:00, ?B/s] 2025-10-10T01:53:05.2872740Z configuration_phimoe.py: 12.3kB [00:00, 44.5MB/s] 2025-10-10T01:53:05.2971084Z A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-3.5-MoE-instruct: 2025-10-10T01:53:05.2971622Z - configuration_phimoe.py 2025-10-10T01:53:05.2972229Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:53:05.3422934Z INFO 10-10 01:53:05 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:53:12.0823057Z INFO 10-10 01:53:12 [model.py:551] Resolved architecture: PhiMoEForCausalLM 2025-10-10T01:53:12.0823533Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:53:12.1069334Z INFO 10-10 01:53:12 [model.py:1545] Using max model len 131072 2025-10-10T01:53:12.1071534Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:53:12.1682309Z INFO 10-10 01:53:12 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:53:12.2113239Z 2025-10-10T01:53:12.2122958Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:12.2123308Z tokenizer_config.json: 3.98kB [00:00, 4.32MB/s] 2025-10-10T01:53:12.3744462Z 2025-10-10T01:53:12.4887718Z tokenizer.model: 0% 0.00/500k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:13.0139463Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:13.0140117Z 2025-10-10T01:53:13.0140539Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:13.0141054Z 2025-10-10T01:53:13.0141739Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:13.0142253Z 2025-10-10T01:53:13.0142558Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:13.0142855Z 2025-10-10T01:53:13.0143143Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0143426Z 2025-10-10T01:53:13.0144172Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:13.0144623Z 2025-10-10T01:53:13.0144962Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:13.0145357Z 2025-10-10T01:53:13.0146117Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:13.0146707Z 2025-10-10T01:53:13.0147086Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:13.0147398Z 2025-10-10T01:53:13.0147683Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0147967Z 2025-10-10T01:53:13.0148548Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:13.0149349Z 2025-10-10T01:53:13.0149806Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] self._init_executor() 2025-10-10T01:53:13.0150307Z 2025-10-10T01:53:13.0151099Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:13.0151800Z 2025-10-10T01:53:13.0152217Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:13.0152547Z 2025-10-10T01:53:13.0153349Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:13.0154010Z 2025-10-10T01:53:13.0154384Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:13.0154717Z 2025-10-10T01:53:13.0155007Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0155296Z 2025-10-10T01:53:13.0155794Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:13.0156372Z 2025-10-10T01:53:13.0156642Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:13.0156927Z 2025-10-10T01:53:13.0157270Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0157532Z 2025-10-10T01:53:13.0158049Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:13.0158527Z 2025-10-10T01:53:13.0158811Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:13.0159219Z 2025-10-10T01:53:13.0159492Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0159765Z 2025-10-10T01:53:13.0160292Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:13.0160770Z 2025-10-10T01:53:13.0161083Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:13.0161439Z 2025-10-10T01:53:13.0161731Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0162016Z 2025-10-10T01:53:13.0162465Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:13.0162880Z 2025-10-10T01:53:13.0163218Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:13.0163552Z 2025-10-10T01:53:13.0163833Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0164112Z 2025-10-10T01:53:13.0164449Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:13.0164776Z 2025-10-10T01:53:13.0165131Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:13.0165465Z 2025-10-10T01:53:13.0165838Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:13.0166192Z 2025-10-10T01:53:13.0166535Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:13.0166865Z 2025-10-10T01:53:13.0167222Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:13.0167575Z 2025-10-10T01:53:13.0167988Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:13.0168352Z 2025-10-10T01:53:13.0168861Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:13.0169327Z 2025-10-10T01:53:13.0169675Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:13.0170012Z 2025-10-10T01:53:13.0170524Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:13.0171041Z 2025-10-10T01:53:13.0171425Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:13.0171817Z 2025-10-10T01:53:13.0172373Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:13.0172874Z 2025-10-10T01:53:13.0173173Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:13.0173462Z 2025-10-10T01:53:13.0174088Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:13.0174658Z 2025-10-10T01:53:13.0175020Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:13.0175371Z 2025-10-10T01:53:13.0175634Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0175907Z 2025-10-10T01:53:13.0176507Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:13.0177022Z 2025-10-10T01:53:13.0177315Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:13.0177606Z 2025-10-10T01:53:13.0177874Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0178143Z 2025-10-10T01:53:13.0178741Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:13.0179282Z 2025-10-10T01:53:13.0179570Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:13.0179856Z 2025-10-10T01:53:13.0180119Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0180380Z 2025-10-10T01:53:13.0180956Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:13.0181487Z 2025-10-10T01:53:13.0181800Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:13.0182119Z 2025-10-10T01:53:13.0182388Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0182655Z 2025-10-10T01:53:13.0183219Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:13.0183702Z 2025-10-10T01:53:13.0183988Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:13.0184268Z 2025-10-10T01:53:13.0184534Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0184796Z 2025-10-10T01:53:13.0185311Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:13.0185833Z 2025-10-10T01:53:13.0186139Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:13.0186470Z 2025-10-10T01:53:13.0186699Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:13.0186983Z 2025-10-10T01:53:13.0187476Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:13.0187930Z 2025-10-10T01:53:13.0188182Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] raise RuntimeError( 2025-10-10T01:53:13.0188437Z 2025-10-10T01:53:13.0189002Z (EngineCore_DP0 pid=8544) ERROR 10-10 01:53:13 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:13.0189696Z (EngineCore_DP0 pid=8544) Process EngineCore_DP0: 2025-10-10T01:53:13.0190081Z (EngineCore_DP0 pid=8544) Traceback (most recent call last): 2025-10-10T01:53:13.0190704Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:13.0191220Z (EngineCore_DP0 pid=8544) self.run() 2025-10-10T01:53:13.0191795Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:13.0192523Z (EngineCore_DP0 pid=8544) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:13.0193342Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:13.0194031Z (EngineCore_DP0 pid=8544) raise e 2025-10-10T01:53:13.0194738Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:13.0195475Z (EngineCore_DP0 pid=8544) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:13.0195940Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0196881Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:13.0197644Z (EngineCore_DP0 pid=8544) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:13.0198413Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:13.0199198Z (EngineCore_DP0 pid=8544) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:13.0199745Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0200461Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:13.0201257Z (EngineCore_DP0 pid=8544) self._init_executor() 2025-10-10T01:53:13.0202092Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:13.0202905Z (EngineCore_DP0 pid=8544) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:13.0203627Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:13.0204336Z (EngineCore_DP0 pid=8544) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:13.0204941Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0205574Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:13.0206227Z (EngineCore_DP0 pid=8544) return func(*args, **kwargs) 2025-10-10T01:53:13.0206619Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0207257Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:13.0207882Z (EngineCore_DP0 pid=8544) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:13.0208300Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0208962Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:13.0209614Z (EngineCore_DP0 pid=8544) module = importlib.import_module(module_name) 2025-10-10T01:53:13.0210055Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0210727Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:13.0211354Z (EngineCore_DP0 pid=8544) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:13.0211840Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0212333Z (EngineCore_DP0 pid=8544) File "", line 1387, in _gcd_import 2025-10-10T01:53:13.0212883Z (EngineCore_DP0 pid=8544) File "", line 1360, in _find_and_load 2025-10-10T01:53:13.0213468Z (EngineCore_DP0 pid=8544) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:13.0214035Z (EngineCore_DP0 pid=8544) File "", line 935, in _load_unlocked 2025-10-10T01:53:13.0214613Z (EngineCore_DP0 pid=8544) File "", line 999, in exec_module 2025-10-10T01:53:13.0215209Z (EngineCore_DP0 pid=8544) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:13.0215938Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:13.0216612Z (EngineCore_DP0 pid=8544) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:13.0217326Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:13.0218037Z (EngineCore_DP0 pid=8544) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:13.0218854Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:13.0219533Z (EngineCore_DP0 pid=8544) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:13.0220329Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:13.0221139Z (EngineCore_DP0 pid=8544) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:13.0221607Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0222341Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:13.0223030Z (EngineCore_DP0 pid=8544) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:13.0223464Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0224261Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:13.0224964Z (EngineCore_DP0 pid=8544) return _is_fa2_supported(device)[0] 2025-10-10T01:53:13.0225368Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0226074Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:13.0226779Z (EngineCore_DP0 pid=8544) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:13.0227227Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0227894Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:13.0228594Z (EngineCore_DP0 pid=8544) prop = get_device_properties(device) 2025-10-10T01:53:13.0229028Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:13.0229689Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:13.0230360Z (EngineCore_DP0 pid=8544) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:13.0230753Z (EngineCore_DP0 pid=8544) ^^^^^^^^^^^^ 2025-10-10T01:53:13.0231332Z (EngineCore_DP0 pid=8544) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:13.0231895Z (EngineCore_DP0 pid=8544) raise RuntimeError( 2025-10-10T01:53:13.0232580Z (EngineCore_DP0 pid=8544) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:13.4165166Z FAILED 2025-10-10T01:53:13.4295327Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForRewardModel] Fork a new process to run a test 8548 2025-10-10T01:53:13.4307617Z Fork a new process to run a test 0 2025-10-10T01:53:13.4311705Z `transformers==4.56.2` installed, but `transformers<=4.53` is required to run this model. Reason: HF model uses remote code that is not compatible with latest Transformers 2025-10-10T01:53:13.7333944Z PASSED 2025-10-10T01:53:13.7463558Z models/test_initialization.py::test_can_initialize_large_subset[GPT2ForSequenceClassification] Fork a new process to run a test 8549 2025-10-10T01:53:13.7474118Z Fork a new process to run a test 0 2025-10-10T01:53:13.7746539Z INFO 10-10 01:53:13 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GPT2ForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nie3e/sentiment-polish-gpt2-small'} 2025-10-10T01:53:13.9185136Z 2025-10-10T01:53:13.9187468Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:13.9187787Z config.json: 1.11kB [00:00, 5.47MB/s] 2025-10-10T01:53:20.6347241Z INFO 10-10 01:53:20 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:53:20.6348039Z INFO 10-10 01:53:20 [model.py:861] Resolved `--convert auto` to `--convert classify`. Pass the value explicitly to silence this message. 2025-10-10T01:53:20.6348907Z INFO 10-10 01:53:20 [model.py:551] Resolved architecture: GPT2ForSequenceClassification 2025-10-10T01:53:20.6349386Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:53:20.6594992Z INFO 10-10 01:53:20 [model.py:1765] Downcasting torch.float32 to torch.float16. 2025-10-10T01:53:20.6595789Z INFO 10-10 01:53:20 [model.py:1545] Using max model len 2048 2025-10-10T01:53:20.6895701Z INFO 10-10 01:53:20 [arg_utils.py:1580] (Enabling) chunked prefill by default 2025-10-10T01:53:20.6896556Z INFO 10-10 01:53:20 [arg_utils.py:1583] (Enabling) prefix caching by default 2025-10-10T01:53:20.8299307Z INFO 10-10 01:53:20 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:53:20.8752617Z 2025-10-10T01:53:20.8754209Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:20.8754564Z tokenizer_config.json: 1.17kB [00:00, 9.16MB/s] 2025-10-10T01:53:20.9462803Z 2025-10-10T01:53:20.9536017Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:20.9536271Z vocab.json: 907kB [00:00, 125MB/s] 2025-10-10T01:53:20.9892076Z 2025-10-10T01:53:20.9918626Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:53:20.9919237Z merges.txt: 559kB [00:00, 215MB/s] 2025-10-10T01:53:21.0324925Z 2025-10-10T01:53:21.0406775Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:21.0407246Z tokenizer.json: 2.34MB [00:00, 287MB/s] 2025-10-10T01:53:21.1083984Z 2025-10-10T01:53:21.1085088Z special_tokens_map.json: 0% 0.00/437 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:21.5286137Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:21.5286593Z 2025-10-10T01:53:21.5287097Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:21.5287799Z 2025-10-10T01:53:21.5288687Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:21.5289473Z 2025-10-10T01:53:21.5289897Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:21.5290246Z 2025-10-10T01:53:21.5290546Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5291049Z 2025-10-10T01:53:21.5291661Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:21.5292174Z 2025-10-10T01:53:21.5292649Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:21.5293089Z 2025-10-10T01:53:21.5293753Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:21.5294226Z 2025-10-10T01:53:21.5294658Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:21.5295226Z 2025-10-10T01:53:21.5295590Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5295893Z 2025-10-10T01:53:21.5297016Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:21.5297700Z 2025-10-10T01:53:21.5298202Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] self._init_executor() 2025-10-10T01:53:21.5298613Z 2025-10-10T01:53:21.5299264Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:21.5299859Z 2025-10-10T01:53:21.5300333Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:21.5300662Z 2025-10-10T01:53:21.5301347Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:21.5301906Z 2025-10-10T01:53:21.5302353Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:21.5302850Z 2025-10-10T01:53:21.5303256Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5303558Z 2025-10-10T01:53:21.5304206Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:21.5304795Z 2025-10-10T01:53:21.5305078Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:21.5305366Z 2025-10-10T01:53:21.5305861Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5306140Z 2025-10-10T01:53:21.5306815Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:21.5307536Z 2025-10-10T01:53:21.5307858Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:21.5308235Z 2025-10-10T01:53:21.5308563Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5308836Z 2025-10-10T01:53:21.5309508Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:21.5310133Z 2025-10-10T01:53:21.5310491Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:21.5310886Z 2025-10-10T01:53:21.5311215Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5311495Z 2025-10-10T01:53:21.5312176Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:21.5312725Z 2025-10-10T01:53:21.5313121Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:21.5313565Z 2025-10-10T01:53:21.5313898Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5314188Z 2025-10-10T01:53:21.5314665Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:21.5315008Z 2025-10-10T01:53:21.5315494Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:21.5315843Z 2025-10-10T01:53:21.5316369Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:21.5316758Z 2025-10-10T01:53:21.5317246Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:21.5317584Z 2025-10-10T01:53:21.5318084Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:21.5318442Z 2025-10-10T01:53:21.5318956Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:21.5319441Z 2025-10-10T01:53:21.5320115Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:21.5320775Z 2025-10-10T01:53:21.5321171Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:21.5321662Z 2025-10-10T01:53:21.5322211Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:21.5322845Z 2025-10-10T01:53:21.5323325Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:21.5323795Z 2025-10-10T01:53:21.5324507Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:21.5325144Z 2025-10-10T01:53:21.5325466Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:21.5325867Z 2025-10-10T01:53:21.5326640Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:21.5327356Z 2025-10-10T01:53:21.5327864Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:21.5328224Z 2025-10-10T01:53:21.5328531Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5328892Z 2025-10-10T01:53:21.5329469Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:21.5330125Z 2025-10-10T01:53:21.5330437Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:21.5330796Z 2025-10-10T01:53:21.5331112Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5331490Z 2025-10-10T01:53:21.5332103Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:21.5332657Z 2025-10-10T01:53:21.5332940Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:21.5333228Z 2025-10-10T01:53:21.5333613Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5333886Z 2025-10-10T01:53:21.5334482Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:21.5335011Z 2025-10-10T01:53:21.5335328Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:21.5335634Z 2025-10-10T01:53:21.5335909Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5336178Z 2025-10-10T01:53:21.5336694Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:21.5337182Z 2025-10-10T01:53:21.5337604Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:21.5337903Z 2025-10-10T01:53:21.5338245Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5338524Z 2025-10-10T01:53:21.5339040Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:21.5339519Z 2025-10-10T01:53:21.5339820Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:21.5340119Z 2025-10-10T01:53:21.5340391Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:21.5340628Z 2025-10-10T01:53:21.5341110Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:21.5341554Z 2025-10-10T01:53:21.5341844Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] raise RuntimeError( 2025-10-10T01:53:21.5342099Z 2025-10-10T01:53:21.5342661Z (EngineCore_DP0 pid=8608) ERROR 10-10 01:53:21 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:21.5343353Z (EngineCore_DP0 pid=8608) Process EngineCore_DP0: 2025-10-10T01:53:21.5343761Z (EngineCore_DP0 pid=8608) Traceback (most recent call last): 2025-10-10T01:53:21.5344408Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:21.5345073Z (EngineCore_DP0 pid=8608) self.run() 2025-10-10T01:53:21.5345603Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:21.5346166Z (EngineCore_DP0 pid=8608) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:21.5346894Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:21.5347443Z (EngineCore_DP0 pid=8608) raise e 2025-10-10T01:53:21.5348033Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:21.5348655Z (EngineCore_DP0 pid=8608) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:21.5349098Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5349726Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:21.5350369Z (EngineCore_DP0 pid=8608) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:21.5351027Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:21.5351645Z (EngineCore_DP0 pid=8608) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:21.5352087Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5352753Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:21.5353333Z (EngineCore_DP0 pid=8608) self._init_executor() 2025-10-10T01:53:21.5353999Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:21.5354682Z (EngineCore_DP0 pid=8608) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:21.5355471Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:21.5356203Z (EngineCore_DP0 pid=8608) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:21.5356697Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5357339Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:21.5358069Z (EngineCore_DP0 pid=8608) return func(*args, **kwargs) 2025-10-10T01:53:21.5358513Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5359212Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:21.5359910Z (EngineCore_DP0 pid=8608) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:21.5360336Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5361014Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:21.5361661Z (EngineCore_DP0 pid=8608) module = importlib.import_module(module_name) 2025-10-10T01:53:21.5362104Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5362682Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:21.5363298Z (EngineCore_DP0 pid=8608) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:21.5363787Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5364329Z (EngineCore_DP0 pid=8608) File "", line 1387, in _gcd_import 2025-10-10T01:53:21.5364888Z (EngineCore_DP0 pid=8608) File "", line 1360, in _find_and_load 2025-10-10T01:53:21.5365472Z (EngineCore_DP0 pid=8608) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:21.5366043Z (EngineCore_DP0 pid=8608) File "", line 935, in _load_unlocked 2025-10-10T01:53:21.5366616Z (EngineCore_DP0 pid=8608) File "", line 999, in exec_module 2025-10-10T01:53:21.5367203Z (EngineCore_DP0 pid=8608) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:21.5367929Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:21.5368602Z (EngineCore_DP0 pid=8608) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:21.5369322Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:21.5370049Z (EngineCore_DP0 pid=8608) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:21.5370826Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:21.5371501Z (EngineCore_DP0 pid=8608) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:21.5372343Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:21.5373160Z (EngineCore_DP0 pid=8608) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:21.5373635Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5374322Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:21.5374996Z (EngineCore_DP0 pid=8608) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:21.5375447Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5376236Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:21.5376929Z (EngineCore_DP0 pid=8608) return _is_fa2_supported(device)[0] 2025-10-10T01:53:21.5377383Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5378092Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:21.5378810Z (EngineCore_DP0 pid=8608) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:21.5379272Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5379935Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:21.5380573Z (EngineCore_DP0 pid=8608) prop = get_device_properties(device) 2025-10-10T01:53:21.5380984Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:21.5381636Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:21.5382364Z (EngineCore_DP0 pid=8608) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:21.5382768Z (EngineCore_DP0 pid=8608) ^^^^^^^^^^^^ 2025-10-10T01:53:21.5383347Z (EngineCore_DP0 pid=8608) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:21.5383917Z (EngineCore_DP0 pid=8608) raise RuntimeError( 2025-10-10T01:53:21.5384772Z (EngineCore_DP0 pid=8608) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:21.9322126Z FAILED 2025-10-10T01:53:21.9451179Z models/test_initialization.py::test_can_initialize_large_subset[ArceeForCausalLM] Fork a new process to run a test 8612 2025-10-10T01:53:21.9462800Z Fork a new process to run a test 0 2025-10-10T01:53:21.9737998Z INFO 10-10 01:53:21 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ArceeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'arcee-ai/AFM-4.5B-Base'} 2025-10-10T01:53:22.1727793Z 2025-10-10T01:53:22.1728742Z config.json: 0% 0.00/843 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:30.5188845Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:30.5189258Z 2025-10-10T01:53:30.5189812Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:30.5190231Z 2025-10-10T01:53:30.5190912Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:30.5191519Z 2025-10-10T01:53:30.5192205Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:30.5192655Z 2025-10-10T01:53:30.5193023Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5193373Z 2025-10-10T01:53:30.5194005Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:30.5194563Z 2025-10-10T01:53:30.5194992Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:30.5195688Z 2025-10-10T01:53:30.5196546Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:30.5197264Z 2025-10-10T01:53:30.5197671Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:30.5197983Z 2025-10-10T01:53:30.5198263Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5198550Z 2025-10-10T01:53:30.5199215Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:30.5200081Z 2025-10-10T01:53:30.5200606Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] self._init_executor() 2025-10-10T01:53:30.5201008Z 2025-10-10T01:53:30.5202045Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:30.5202979Z 2025-10-10T01:53:30.5203793Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:30.5204317Z 2025-10-10T01:53:30.5204935Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:30.5205460Z 2025-10-10T01:53:30.5205820Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:30.5206163Z 2025-10-10T01:53:30.5206453Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5206738Z 2025-10-10T01:53:30.5207224Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:30.5207704Z 2025-10-10T01:53:30.5207968Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:30.5208248Z 2025-10-10T01:53:30.5208495Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5208760Z 2025-10-10T01:53:30.5209273Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:30.5209747Z 2025-10-10T01:53:30.5210038Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:30.5210331Z 2025-10-10T01:53:30.5210601Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5210871Z 2025-10-10T01:53:30.5211503Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:30.5212024Z 2025-10-10T01:53:30.5212356Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:30.5212667Z 2025-10-10T01:53:30.5212938Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5213217Z 2025-10-10T01:53:30.5213731Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:30.5214146Z 2025-10-10T01:53:30.5214487Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:30.5214887Z 2025-10-10T01:53:30.5215187Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5215461Z 2025-10-10T01:53:30.5215818Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:30.5216156Z 2025-10-10T01:53:30.5216512Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:30.5216855Z 2025-10-10T01:53:30.5217242Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:30.5217604Z 2025-10-10T01:53:30.5217942Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:30.5218309Z 2025-10-10T01:53:30.5218737Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:30.5219097Z 2025-10-10T01:53:30.5219479Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:30.5219858Z 2025-10-10T01:53:30.5220374Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:30.5220846Z 2025-10-10T01:53:30.5221193Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:30.5221525Z 2025-10-10T01:53:30.5222054Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:30.5222534Z 2025-10-10T01:53:30.5222913Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:30.5223268Z 2025-10-10T01:53:30.5223829Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:30.5224339Z 2025-10-10T01:53:30.5224629Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:30.5224942Z 2025-10-10T01:53:30.5225570Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:30.5226148Z 2025-10-10T01:53:30.5226567Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:30.5226931Z 2025-10-10T01:53:30.5227185Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5227452Z 2025-10-10T01:53:30.5228012Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:30.5228536Z 2025-10-10T01:53:30.5228867Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:30.5229167Z 2025-10-10T01:53:30.5229470Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5229748Z 2025-10-10T01:53:30.5230417Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:30.5230968Z 2025-10-10T01:53:30.5231253Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:30.5231534Z 2025-10-10T01:53:30.5231798Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5232061Z 2025-10-10T01:53:30.5232642Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:30.5233171Z 2025-10-10T01:53:30.5233486Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:30.5233816Z 2025-10-10T01:53:30.5234137Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5234418Z 2025-10-10T01:53:30.5234942Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:30.5235430Z 2025-10-10T01:53:30.5235706Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:30.5236006Z 2025-10-10T01:53:30.5236305Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5236569Z 2025-10-10T01:53:30.5237094Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:30.5237597Z 2025-10-10T01:53:30.5237909Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:30.5238213Z 2025-10-10T01:53:30.5238444Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:30.5238683Z 2025-10-10T01:53:30.5239286Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:30.5239753Z 2025-10-10T01:53:30.5240002Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] raise RuntimeError( 2025-10-10T01:53:30.5240265Z 2025-10-10T01:53:30.5240829Z (EngineCore_DP0 pid=8690) ERROR 10-10 01:53:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:30.5241586Z (EngineCore_DP0 pid=8690) Process EngineCore_DP0: 2025-10-10T01:53:30.5241996Z (EngineCore_DP0 pid=8690) Traceback (most recent call last): 2025-10-10T01:53:30.5242603Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:30.5243111Z (EngineCore_DP0 pid=8690) self.run() 2025-10-10T01:53:30.5243641Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:30.5244204Z (EngineCore_DP0 pid=8690) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:30.5244937Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:30.5245479Z (EngineCore_DP0 pid=8690) raise e 2025-10-10T01:53:30.5246061Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:30.5246787Z (EngineCore_DP0 pid=8690) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:30.5247226Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5247840Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:30.5248488Z (EngineCore_DP0 pid=8690) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:30.5249143Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:30.5249761Z (EngineCore_DP0 pid=8690) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:30.5250209Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5250908Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:30.5251508Z (EngineCore_DP0 pid=8690) self._init_executor() 2025-10-10T01:53:30.5252189Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:30.5252887Z (EngineCore_DP0 pid=8690) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:30.5253613Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:30.5254321Z (EngineCore_DP0 pid=8690) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:30.5254830Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5255470Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:30.5256051Z (EngineCore_DP0 pid=8690) return func(*args, **kwargs) 2025-10-10T01:53:30.5256434Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5257069Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:30.5257710Z (EngineCore_DP0 pid=8690) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:30.5258134Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5258850Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:30.5259521Z (EngineCore_DP0 pid=8690) module = importlib.import_module(module_name) 2025-10-10T01:53:30.5259967Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5260544Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:30.5261169Z (EngineCore_DP0 pid=8690) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:30.5261653Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5262221Z (EngineCore_DP0 pid=8690) File "", line 1387, in _gcd_import 2025-10-10T01:53:30.5262778Z (EngineCore_DP0 pid=8690) File "", line 1360, in _find_and_load 2025-10-10T01:53:30.5263434Z (EngineCore_DP0 pid=8690) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:30.5264018Z (EngineCore_DP0 pid=8690) File "", line 935, in _load_unlocked 2025-10-10T01:53:30.5264583Z (EngineCore_DP0 pid=8690) File "", line 999, in exec_module 2025-10-10T01:53:30.5265181Z (EngineCore_DP0 pid=8690) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:30.5265911Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:30.5266592Z (EngineCore_DP0 pid=8690) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:30.5267313Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:30.5268090Z (EngineCore_DP0 pid=8690) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:30.5268878Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:30.5269547Z (EngineCore_DP0 pid=8690) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:30.5270338Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:30.5271153Z (EngineCore_DP0 pid=8690) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:30.5271629Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5272316Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:30.5273013Z (EngineCore_DP0 pid=8690) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:30.5273452Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5274193Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:30.5274887Z (EngineCore_DP0 pid=8690) return _is_fa2_supported(device)[0] 2025-10-10T01:53:30.5275306Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5276015Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:30.5276769Z (EngineCore_DP0 pid=8690) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:30.5277243Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5277913Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:30.5278544Z (EngineCore_DP0 pid=8690) prop = get_device_properties(device) 2025-10-10T01:53:30.5278950Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:30.5279691Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:30.5280425Z (EngineCore_DP0 pid=8690) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:30.5280827Z (EngineCore_DP0 pid=8690) ^^^^^^^^^^^^ 2025-10-10T01:53:30.5281401Z (EngineCore_DP0 pid=8690) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:30.5282015Z (EngineCore_DP0 pid=8690) raise RuntimeError( 2025-10-10T01:53:30.5282698Z (EngineCore_DP0 pid=8690) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:30.9449631Z FAILED 2025-10-10T01:53:30.9579416Z models/test_initialization.py::test_can_initialize_large_subset[Zamba2ForCausalLM] Fork a new process to run a test 8694 2025-10-10T01:53:30.9590487Z Fork a new process to run a test 0 2025-10-10T01:53:30.9866442Z INFO 10-10 01:53:30 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Zamba2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Zyphra/Zamba2-7B-instruct'} 2025-10-10T01:53:31.2072979Z 2025-10-10T01:53:31.2075344Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:31.2075911Z config.json: 2.41kB [00:00, 12.2MB/s] 2025-10-10T01:53:38.0921657Z INFO 10-10 01:53:38 [model.py:551] Resolved architecture: Zamba2ForCausalLM 2025-10-10T01:53:38.0922157Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:53:38.2629536Z 2025-10-10T01:53:38.2632776Z model.safetensors.index.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:38.2633198Z model.safetensors.index.json: 63.3kB [00:00, 215MB/s] 2025-10-10T01:53:38.2669199Z 2025-10-10T01:53:38.3704755Z Parse safetensors files: 0% 0/3 [00:00= mamba page size. 2025-10-10T01:53:38.6381354Z INFO 10-10 01:53:38 [config.py:397] Padding mamba page size by 43.05% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:53:38.7065492Z 2025-10-10T01:53:38.7067260Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:38.7067612Z tokenizer_config.json: 1.37kB [00:00, 6.63MB/s] 2025-10-10T01:53:38.9068674Z 2025-10-10T01:53:38.9161400Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:53:38.9161918Z tokenizer.json: 1.80MB [00:00, 194MB/s] 2025-10-10T01:53:39.0378423Z 2025-10-10T01:53:39.0379692Z special_tokens_map.json: 0% 0.00/504 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:39.4110861Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:39.4111427Z 2025-10-10T01:53:39.4111857Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:39.4112225Z 2025-10-10T01:53:39.4112905Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:39.4113501Z 2025-10-10T01:53:39.4113909Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:39.4114301Z 2025-10-10T01:53:39.4114657Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4115020Z 2025-10-10T01:53:39.4115647Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:39.4116114Z 2025-10-10T01:53:39.4116618Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:39.4116964Z 2025-10-10T01:53:39.4117455Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:39.4117897Z 2025-10-10T01:53:39.4118381Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:39.4118704Z 2025-10-10T01:53:39.4119102Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4119532Z 2025-10-10T01:53:39.4120070Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:39.4120545Z 2025-10-10T01:53:39.4120805Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] self._init_executor() 2025-10-10T01:53:39.4121699Z 2025-10-10T01:53:39.4122296Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:39.4123244Z 2025-10-10T01:53:39.4123846Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:39.4124454Z 2025-10-10T01:53:39.4125522Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:39.4126461Z 2025-10-10T01:53:39.4126865Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:39.4127219Z 2025-10-10T01:53:39.4127638Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4127939Z 2025-10-10T01:53:39.4128438Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:39.4128918Z 2025-10-10T01:53:39.4129184Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:39.4129458Z 2025-10-10T01:53:39.4129706Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4129974Z 2025-10-10T01:53:39.4130487Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:39.4130970Z 2025-10-10T01:53:39.4131256Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:39.4131548Z 2025-10-10T01:53:39.4131817Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4132081Z 2025-10-10T01:53:39.4132627Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:39.4133137Z 2025-10-10T01:53:39.4133449Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:39.4133751Z 2025-10-10T01:53:39.4134023Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4134302Z 2025-10-10T01:53:39.4134790Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:39.4135214Z 2025-10-10T01:53:39.4135560Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:39.4135904Z 2025-10-10T01:53:39.4136189Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4136464Z 2025-10-10T01:53:39.4136810Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:39.4137183Z 2025-10-10T01:53:39.4137559Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:39.4137890Z 2025-10-10T01:53:39.4138273Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:39.4138678Z 2025-10-10T01:53:39.4139019Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:39.4139364Z 2025-10-10T01:53:39.4139722Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:39.4140077Z 2025-10-10T01:53:39.4140466Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:39.4140837Z 2025-10-10T01:53:39.4141334Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:39.4141826Z 2025-10-10T01:53:39.4142233Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:39.4142576Z 2025-10-10T01:53:39.4143108Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:39.4143586Z 2025-10-10T01:53:39.4143963Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:39.4144322Z 2025-10-10T01:53:39.4144882Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:39.4145393Z 2025-10-10T01:53:39.4145697Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:39.4145994Z 2025-10-10T01:53:39.4146622Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:39.4147206Z 2025-10-10T01:53:39.4147572Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:39.4147929Z 2025-10-10T01:53:39.4148182Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4148454Z 2025-10-10T01:53:39.4149008Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:39.4149530Z 2025-10-10T01:53:39.4149876Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:39.4150185Z 2025-10-10T01:53:39.4150480Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4150767Z 2025-10-10T01:53:39.4151393Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:39.4151948Z 2025-10-10T01:53:39.4152237Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:39.4152575Z 2025-10-10T01:53:39.4152838Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4153110Z 2025-10-10T01:53:39.4153694Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:39.4154268Z 2025-10-10T01:53:39.4154594Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:39.4154910Z 2025-10-10T01:53:39.4155185Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4155469Z 2025-10-10T01:53:39.4155988Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:39.4156477Z 2025-10-10T01:53:39.4156761Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:39.4157043Z 2025-10-10T01:53:39.4157309Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4157575Z 2025-10-10T01:53:39.4158155Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:39.4158642Z 2025-10-10T01:53:39.4158960Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:39.4159373Z 2025-10-10T01:53:39.4159600Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:39.4159852Z 2025-10-10T01:53:39.4160358Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:39.4160830Z 2025-10-10T01:53:39.4161080Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] raise RuntimeError( 2025-10-10T01:53:39.4161348Z 2025-10-10T01:53:39.4161917Z (EngineCore_DP0 pid=8755) ERROR 10-10 01:53:39 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:39.4162622Z (EngineCore_DP0 pid=8755) Process EngineCore_DP0: 2025-10-10T01:53:39.4163015Z (EngineCore_DP0 pid=8755) Traceback (most recent call last): 2025-10-10T01:53:39.4163618Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:39.4164137Z (EngineCore_DP0 pid=8755) self.run() 2025-10-10T01:53:39.4164666Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:39.4165282Z (EngineCore_DP0 pid=8755) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:39.4165975Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:39.4166526Z (EngineCore_DP0 pid=8755) raise e 2025-10-10T01:53:39.4167124Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:39.4167756Z (EngineCore_DP0 pid=8755) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:39.4168200Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4168885Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:39.4169519Z (EngineCore_DP0 pid=8755) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:39.4170267Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:39.4170885Z (EngineCore_DP0 pid=8755) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:39.4171335Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4171983Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:39.4172559Z (EngineCore_DP0 pid=8755) self._init_executor() 2025-10-10T01:53:39.4173229Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:39.4173921Z (EngineCore_DP0 pid=8755) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:39.4174696Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:39.4175432Z (EngineCore_DP0 pid=8755) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:39.4175921Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4176564Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:39.4177152Z (EngineCore_DP0 pid=8755) return func(*args, **kwargs) 2025-10-10T01:53:39.4177534Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4178170Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:39.4178809Z (EngineCore_DP0 pid=8755) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:39.4179231Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4179894Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:39.4180546Z (EngineCore_DP0 pid=8755) module = importlib.import_module(module_name) 2025-10-10T01:53:39.4180986Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4181564Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:39.4182182Z (EngineCore_DP0 pid=8755) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:39.4182718Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4183237Z (EngineCore_DP0 pid=8755) File "", line 1387, in _gcd_import 2025-10-10T01:53:39.4183792Z (EngineCore_DP0 pid=8755) File "", line 1360, in _find_and_load 2025-10-10T01:53:39.4184378Z (EngineCore_DP0 pid=8755) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:39.4184950Z (EngineCore_DP0 pid=8755) File "", line 935, in _load_unlocked 2025-10-10T01:53:39.4185525Z (EngineCore_DP0 pid=8755) File "", line 999, in exec_module 2025-10-10T01:53:39.4186188Z (EngineCore_DP0 pid=8755) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:39.4186917Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:39.4187641Z (EngineCore_DP0 pid=8755) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:39.4188360Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:39.4189087Z (EngineCore_DP0 pid=8755) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:39.4189866Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:39.4190538Z (EngineCore_DP0 pid=8755) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:39.4191331Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:39.4192150Z (EngineCore_DP0 pid=8755) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:39.4192683Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4193386Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:39.4194069Z (EngineCore_DP0 pid=8755) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:39.4194512Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4195258Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:39.4195957Z (EngineCore_DP0 pid=8755) return _is_fa2_supported(device)[0] 2025-10-10T01:53:39.4196749Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4197473Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:39.4198183Z (EngineCore_DP0 pid=8755) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:39.4198633Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4199379Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:39.4200043Z (EngineCore_DP0 pid=8755) prop = get_device_properties(device) 2025-10-10T01:53:39.4200463Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:39.4201212Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:39.4201900Z (EngineCore_DP0 pid=8755) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:39.4202293Z (EngineCore_DP0 pid=8755) ^^^^^^^^^^^^ 2025-10-10T01:53:39.4202867Z (EngineCore_DP0 pid=8755) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:39.4203425Z (EngineCore_DP0 pid=8755) raise RuntimeError( 2025-10-10T01:53:39.4204107Z (EngineCore_DP0 pid=8755) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:39.8204830Z FAILED 2025-10-10T01:53:39.8334993Z models/test_initialization.py::test_can_initialize_large_subset[Step3VLForConditionalGeneration] Fork a new process to run a test 8759 2025-10-10T01:53:39.8346473Z Fork a new process to run a test 0 2025-10-10T01:53:39.8618368Z INFO 10-10 01:53:39 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Step3VLForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'stepfun-ai/step3'} 2025-10-10T01:53:40.2391902Z INFO 10-10 01:53:40 [model.py:551] Resolved architecture: Step3VLForConditionalGeneration 2025-10-10T01:53:40.2392575Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:53:40.2640394Z INFO 10-10 01:53:40 [model.py:1545] Using max model len 65536 2025-10-10T01:53:40.2643141Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:53:40.4406822Z INFO 10-10 01:53:40 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:53:41.3784544Z (EngineCore_DP0 pid=8766) INFO 10-10 01:53:41 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:53:41.3860107Z (EngineCore_DP0 pid=8766) INFO 10-10 01:53:41 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='stepfun-ai/step3', speculative_config=None, tokenizer='stepfun-ai/step3', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=stepfun-ai/step3, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:41.5213918Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:41.5214497Z 2025-10-10T01:53:41.5215123Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:41.5215516Z 2025-10-10T01:53:41.5216383Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:41.5217305Z 2025-10-10T01:53:41.5217714Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:41.5218115Z 2025-10-10T01:53:41.5218408Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5218694Z 2025-10-10T01:53:41.5219211Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:41.5219663Z 2025-10-10T01:53:41.5219989Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:41.5220360Z 2025-10-10T01:53:41.5221011Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:41.5221603Z 2025-10-10T01:53:41.5221949Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:41.5222417Z 2025-10-10T01:53:41.5222835Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5223147Z 2025-10-10T01:53:41.5223669Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:41.5224148Z 2025-10-10T01:53:41.5224524Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] self._init_executor() 2025-10-10T01:53:41.5224999Z 2025-10-10T01:53:41.5226001Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:41.5226742Z 2025-10-10T01:53:41.5227255Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:41.5227698Z 2025-10-10T01:53:41.5228688Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:41.5229389Z 2025-10-10T01:53:41.5229888Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:41.5230239Z 2025-10-10T01:53:41.5230537Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5230826Z 2025-10-10T01:53:41.5231327Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:41.5231787Z 2025-10-10T01:53:41.5232175Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:41.5232465Z 2025-10-10T01:53:41.5232730Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5232991Z 2025-10-10T01:53:41.5233521Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:41.5233996Z 2025-10-10T01:53:41.5234286Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:41.5234637Z 2025-10-10T01:53:41.5234906Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5235182Z 2025-10-10T01:53:41.5235706Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:41.5236258Z 2025-10-10T01:53:41.5236568Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:41.5236873Z 2025-10-10T01:53:41.5237145Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5237424Z 2025-10-10T01:53:41.5237868Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:41.5238289Z 2025-10-10T01:53:41.5238635Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:41.5238964Z 2025-10-10T01:53:41.5239394Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5239684Z 2025-10-10T01:53:41.5240106Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:41.5240442Z 2025-10-10T01:53:41.5240805Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:41.5241148Z 2025-10-10T01:53:41.5241522Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:41.5241889Z 2025-10-10T01:53:41.5242229Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:41.5242566Z 2025-10-10T01:53:41.5242926Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:41.5243299Z 2025-10-10T01:53:41.5243677Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:41.5244040Z 2025-10-10T01:53:41.5244550Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:41.5245009Z 2025-10-10T01:53:41.5245363Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:41.5245697Z 2025-10-10T01:53:41.5246217Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:41.5246695Z 2025-10-10T01:53:41.5247123Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:41.5247496Z 2025-10-10T01:53:41.5248047Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:41.5248554Z 2025-10-10T01:53:41.5248843Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:41.5249149Z 2025-10-10T01:53:41.5249807Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:41.5250435Z 2025-10-10T01:53:41.5250804Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:41.5251203Z 2025-10-10T01:53:41.5251466Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5251735Z 2025-10-10T01:53:41.5252299Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:41.5252809Z 2025-10-10T01:53:41.5253108Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:41.5253398Z 2025-10-10T01:53:41.5253685Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5253955Z 2025-10-10T01:53:41.5254567Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:41.5255121Z 2025-10-10T01:53:41.5255447Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:41.5255738Z 2025-10-10T01:53:41.5255998Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5256267Z 2025-10-10T01:53:41.5256854Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:41.5257391Z 2025-10-10T01:53:41.5257705Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:41.5258013Z 2025-10-10T01:53:41.5258288Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5258562Z 2025-10-10T01:53:41.5259109Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:41.5259596Z 2025-10-10T01:53:41.5259880Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:41.5260162Z 2025-10-10T01:53:41.5260433Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5260712Z 2025-10-10T01:53:41.5261232Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:41.5261716Z 2025-10-10T01:53:41.5262020Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:41.5262372Z 2025-10-10T01:53:41.5262603Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:41.5262845Z 2025-10-10T01:53:41.5263333Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:41.5263781Z 2025-10-10T01:53:41.5264027Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] raise RuntimeError( 2025-10-10T01:53:41.5264282Z 2025-10-10T01:53:41.5264851Z (EngineCore_DP0 pid=8766) ERROR 10-10 01:53:41 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:41.5265631Z (EngineCore_DP0 pid=8766) Process EngineCore_DP0: 2025-10-10T01:53:41.5266026Z (EngineCore_DP0 pid=8766) Traceback (most recent call last): 2025-10-10T01:53:41.5266687Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:41.5267203Z (EngineCore_DP0 pid=8766) self.run() 2025-10-10T01:53:41.5267727Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:41.5268289Z (EngineCore_DP0 pid=8766) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:41.5268968Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:41.5269511Z (EngineCore_DP0 pid=8766) raise e 2025-10-10T01:53:41.5270094Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:41.5270725Z (EngineCore_DP0 pid=8766) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:41.5271243Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5271893Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:41.5272534Z (EngineCore_DP0 pid=8766) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:41.5273192Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:41.5273813Z (EngineCore_DP0 pid=8766) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:41.5274268Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5274918Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:41.5275503Z (EngineCore_DP0 pid=8766) self._init_executor() 2025-10-10T01:53:41.5276178Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:41.5276867Z (EngineCore_DP0 pid=8766) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:41.5277590Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:41.5278297Z (EngineCore_DP0 pid=8766) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:41.5278791Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5279536Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:41.5280157Z (EngineCore_DP0 pid=8766) return func(*args, **kwargs) 2025-10-10T01:53:41.5280568Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5281211Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:41.5281843Z (EngineCore_DP0 pid=8766) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:41.5282263Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5282977Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:41.5283633Z (EngineCore_DP0 pid=8766) module = importlib.import_module(module_name) 2025-10-10T01:53:41.5284138Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5284724Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:41.5285357Z (EngineCore_DP0 pid=8766) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:41.5285846Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5286344Z (EngineCore_DP0 pid=8766) File "", line 1387, in _gcd_import 2025-10-10T01:53:41.5286915Z (EngineCore_DP0 pid=8766) File "", line 1360, in _find_and_load 2025-10-10T01:53:41.5287506Z (EngineCore_DP0 pid=8766) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:41.5288088Z (EngineCore_DP0 pid=8766) File "", line 935, in _load_unlocked 2025-10-10T01:53:41.5288711Z (EngineCore_DP0 pid=8766) File "", line 999, in exec_module 2025-10-10T01:53:41.5289326Z (EngineCore_DP0 pid=8766) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:41.5290113Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:41.5290789Z (EngineCore_DP0 pid=8766) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:41.5291512Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:41.5292242Z (EngineCore_DP0 pid=8766) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:41.5293031Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:41.5293705Z (EngineCore_DP0 pid=8766) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:41.5294487Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:41.5295294Z (EngineCore_DP0 pid=8766) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:41.5295781Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5296864Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:41.5297585Z (EngineCore_DP0 pid=8766) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:41.5298139Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5298905Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:41.5299600Z (EngineCore_DP0 pid=8766) return _is_fa2_supported(device)[0] 2025-10-10T01:53:41.5300007Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5300718Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:41.5301513Z (EngineCore_DP0 pid=8766) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:41.5301964Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5302705Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:41.5303341Z (EngineCore_DP0 pid=8766) prop = get_device_properties(device) 2025-10-10T01:53:41.5303749Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:41.5304406Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:41.5305068Z (EngineCore_DP0 pid=8766) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:41.5305477Z (EngineCore_DP0 pid=8766) ^^^^^^^^^^^^ 2025-10-10T01:53:41.5306062Z (EngineCore_DP0 pid=8766) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:41.5306622Z (EngineCore_DP0 pid=8766) raise RuntimeError( 2025-10-10T01:53:41.5307386Z (EngineCore_DP0 pid=8766) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:41.9434906Z FAILED 2025-10-10T01:53:41.9564685Z models/test_initialization.py::test_can_initialize_large_subset[Llama4ForCausalLM] Fork a new process to run a test 8770 2025-10-10T01:53:41.9576541Z Fork a new process to run a test 0 2025-10-10T01:53:41.9578405Z Model is not available online 2025-10-10T01:53:42.2626113Z PASSED 2025-10-10T01:53:42.2756441Z models/test_initialization.py::test_can_initialize_large_subset[GPTBigCodeForCausalLM] Fork a new process to run a test 8771 2025-10-10T01:53:42.2768082Z Fork a new process to run a test 0 2025-10-10T01:53:42.3042273Z INFO 10-10 01:53:42 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GPTBigCodeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'bigcode/starcoder'} 2025-10-10T01:53:42.5580872Z 2025-10-10T01:53:42.5581512Z config.json: 0% 0.00/1.05k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:50.4285123Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:50.4285690Z 2025-10-10T01:53:50.4286087Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:50.4286584Z 2025-10-10T01:53:50.4287324Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:50.4287915Z 2025-10-10T01:53:50.4288307Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:50.4288911Z 2025-10-10T01:53:50.4289307Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4289658Z 2025-10-10T01:53:50.4290281Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:50.4290846Z 2025-10-10T01:53:50.4291519Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:50.4292309Z 2025-10-10T01:53:50.4293144Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:50.4293746Z 2025-10-10T01:53:50.4294164Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:50.4294702Z 2025-10-10T01:53:50.4295075Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4295533Z 2025-10-10T01:53:50.4296701Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:50.4297586Z 2025-10-10T01:53:50.4298092Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] self._init_executor() 2025-10-10T01:53:50.4298585Z 2025-10-10T01:53:50.4299642Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:50.4300334Z 2025-10-10T01:53:50.4300723Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:50.4301074Z 2025-10-10T01:53:50.4301752Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:50.4302289Z 2025-10-10T01:53:50.4302640Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:50.4302978Z 2025-10-10T01:53:50.4303264Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4303551Z 2025-10-10T01:53:50.4304046Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:50.4304504Z 2025-10-10T01:53:50.4304770Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:50.4305042Z 2025-10-10T01:53:50.4305302Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4305566Z 2025-10-10T01:53:50.4306103Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:50.4306601Z 2025-10-10T01:53:50.4306900Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:50.4307192Z 2025-10-10T01:53:50.4307475Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4307750Z 2025-10-10T01:53:50.4308349Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:50.4308846Z 2025-10-10T01:53:50.4309154Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:50.4309460Z 2025-10-10T01:53:50.4309734Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4310005Z 2025-10-10T01:53:50.4310497Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:50.4310988Z 2025-10-10T01:53:50.4311343Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:50.4311679Z 2025-10-10T01:53:50.4311968Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4312318Z 2025-10-10T01:53:50.4312662Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:50.4312991Z 2025-10-10T01:53:50.4313347Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:50.4313680Z 2025-10-10T01:53:50.4314054Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:50.4314413Z 2025-10-10T01:53:50.4314765Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:50.4315099Z 2025-10-10T01:53:50.4315471Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:50.4315824Z 2025-10-10T01:53:50.4316249Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:50.4316615Z 2025-10-10T01:53:50.4317109Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:50.4317569Z 2025-10-10T01:53:50.4317924Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:50.4318268Z 2025-10-10T01:53:50.4318783Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:50.4319393Z 2025-10-10T01:53:50.4319774Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:50.4320138Z 2025-10-10T01:53:50.4320693Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:50.4321194Z 2025-10-10T01:53:50.4321489Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:50.4321782Z 2025-10-10T01:53:50.4322404Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:50.4322971Z 2025-10-10T01:53:50.4323394Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:50.4323752Z 2025-10-10T01:53:50.4324011Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4324276Z 2025-10-10T01:53:50.4324834Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:50.4325352Z 2025-10-10T01:53:50.4325646Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:50.4326004Z 2025-10-10T01:53:50.4326274Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4326544Z 2025-10-10T01:53:50.4327145Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:50.4327742Z 2025-10-10T01:53:50.4328028Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:50.4328310Z 2025-10-10T01:53:50.4328571Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4328834Z 2025-10-10T01:53:50.4329413Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:50.4329937Z 2025-10-10T01:53:50.4330251Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:50.4330589Z 2025-10-10T01:53:50.4330867Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4331161Z 2025-10-10T01:53:50.4331747Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:50.4332243Z 2025-10-10T01:53:50.4332524Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:50.4332803Z 2025-10-10T01:53:50.4333068Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4333336Z 2025-10-10T01:53:50.4333858Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:50.4334333Z 2025-10-10T01:53:50.4334644Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:50.4334960Z 2025-10-10T01:53:50.4335196Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:50.4335432Z 2025-10-10T01:53:50.4335912Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:50.4336366Z 2025-10-10T01:53:50.4336607Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] raise RuntimeError( 2025-10-10T01:53:50.4336867Z 2025-10-10T01:53:50.4337433Z (EngineCore_DP0 pid=8829) ERROR 10-10 01:53:50 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:50.4338126Z (EngineCore_DP0 pid=8829) Process EngineCore_DP0: 2025-10-10T01:53:50.4338591Z (EngineCore_DP0 pid=8829) Traceback (most recent call last): 2025-10-10T01:53:50.4339210Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:50.4339717Z (EngineCore_DP0 pid=8829) self.run() 2025-10-10T01:53:50.4340257Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:50.4340818Z (EngineCore_DP0 pid=8829) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:50.4341489Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:50.4342082Z (EngineCore_DP0 pid=8829) raise e 2025-10-10T01:53:50.4342662Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:50.4343375Z (EngineCore_DP0 pid=8829) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:50.4343822Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4344451Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:50.4345090Z (EngineCore_DP0 pid=8829) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:50.4345755Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:50.4346375Z (EngineCore_DP0 pid=8829) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:50.4346829Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4347480Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:50.4348130Z (EngineCore_DP0 pid=8829) self._init_executor() 2025-10-10T01:53:50.4348793Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:50.4349481Z (EngineCore_DP0 pid=8829) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:50.4350197Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:50.4350923Z (EngineCore_DP0 pid=8829) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:50.4351413Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4352044Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:50.4352630Z (EngineCore_DP0 pid=8829) return func(*args, **kwargs) 2025-10-10T01:53:50.4353011Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4353643Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:50.4354267Z (EngineCore_DP0 pid=8829) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:50.4354685Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4355348Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:50.4356074Z (EngineCore_DP0 pid=8829) module = importlib.import_module(module_name) 2025-10-10T01:53:50.4356531Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4357108Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:50.4357733Z (EngineCore_DP0 pid=8829) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:50.4358229Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4358724Z (EngineCore_DP0 pid=8829) File "", line 1387, in _gcd_import 2025-10-10T01:53:50.4359431Z (EngineCore_DP0 pid=8829) File "", line 1360, in _find_and_load 2025-10-10T01:53:50.4360019Z (EngineCore_DP0 pid=8829) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:50.4360646Z (EngineCore_DP0 pid=8829) File "", line 935, in _load_unlocked 2025-10-10T01:53:50.4361231Z (EngineCore_DP0 pid=8829) File "", line 999, in exec_module 2025-10-10T01:53:50.4361832Z (EngineCore_DP0 pid=8829) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:50.4362565Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:50.4363245Z (EngineCore_DP0 pid=8829) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:50.4363965Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:50.4364682Z (EngineCore_DP0 pid=8829) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:50.4365517Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:50.4366197Z (EngineCore_DP0 pid=8829) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:50.4366989Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:50.4367795Z (EngineCore_DP0 pid=8829) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:50.4368273Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4368970Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:50.4369654Z (EngineCore_DP0 pid=8829) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:50.4370102Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4370856Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:50.4371551Z (EngineCore_DP0 pid=8829) return _is_fa2_supported(device)[0] 2025-10-10T01:53:50.4371954Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4372675Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:50.4373388Z (EngineCore_DP0 pid=8829) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:50.4373890Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4374573Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:50.4375202Z (EngineCore_DP0 pid=8829) prop = get_device_properties(device) 2025-10-10T01:53:50.4375620Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:50.4376273Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:50.4376976Z (EngineCore_DP0 pid=8829) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:50.4377369Z (EngineCore_DP0 pid=8829) ^^^^^^^^^^^^ 2025-10-10T01:53:50.4377938Z (EngineCore_DP0 pid=8829) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:50.4378540Z (EngineCore_DP0 pid=8829) raise RuntimeError( 2025-10-10T01:53:50.4379227Z (EngineCore_DP0 pid=8829) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:50.8301321Z FAILED 2025-10-10T01:53:50.8430084Z models/test_initialization.py::test_can_initialize_large_subset[GlmForCausalLM] Fork a new process to run a test 8833 2025-10-10T01:53:50.8442615Z Fork a new process to run a test 0 2025-10-10T01:53:50.8714935Z INFO 10-10 01:53:50 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GlmForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'zai-org/glm-4-9b-chat-hf'} 2025-10-10T01:53:50.9931878Z 2025-10-10T01:53:50.9932847Z config.json: 0% 0.00/661 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:53:59.5401262Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] EngineCore failed to start. 2025-10-10T01:53:59.5401942Z 2025-10-10T01:53:59.5402473Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] Traceback (most recent call last): 2025-10-10T01:53:59.5402890Z 2025-10-10T01:53:59.5403575Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:59.5404190Z 2025-10-10T01:53:59.5404533Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:59.5405132Z 2025-10-10T01:53:59.5405554Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5405844Z 2025-10-10T01:53:59.5406347Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:59.5406792Z 2025-10-10T01:53:59.5407132Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:59.5407468Z 2025-10-10T01:53:59.5407955Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:59.5408457Z 2025-10-10T01:53:59.5409064Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:59.5409634Z 2025-10-10T01:53:59.5410142Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5410660Z 2025-10-10T01:53:59.5411220Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:59.5411710Z 2025-10-10T01:53:59.5412172Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] self._init_executor() 2025-10-10T01:53:59.5412461Z 2025-10-10T01:53:59.5413151Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:59.5413682Z 2025-10-10T01:53:59.5414157Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:59.5414506Z 2025-10-10T01:53:59.5415057Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:59.5415553Z 2025-10-10T01:53:59.5415912Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:59.5416345Z 2025-10-10T01:53:59.5416641Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5416917Z 2025-10-10T01:53:59.5417411Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:59.5417935Z 2025-10-10T01:53:59.5418197Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:53:59.5418467Z 2025-10-10T01:53:59.5418712Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5418973Z 2025-10-10T01:53:59.5419498Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:59.5419988Z 2025-10-10T01:53:59.5420275Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:59.5420564Z 2025-10-10T01:53:59.5420826Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5421093Z 2025-10-10T01:53:59.5421669Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:59.5422154Z 2025-10-10T01:53:59.5422464Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:53:59.5422771Z 2025-10-10T01:53:59.5423047Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5423317Z 2025-10-10T01:53:59.5423756Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:59.5424179Z 2025-10-10T01:53:59.5424556Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:59.5424883Z 2025-10-10T01:53:59.5425174Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5425447Z 2025-10-10T01:53:59.5425789Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:53:59.5426112Z 2025-10-10T01:53:59.5426450Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:53:59.5426787Z 2025-10-10T01:53:59.5427158Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:59.5427516Z 2025-10-10T01:53:59.5427850Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:53:59.5428192Z 2025-10-10T01:53:59.5428603Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:53:59.5428970Z 2025-10-10T01:53:59.5429341Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:59.5429698Z 2025-10-10T01:53:59.5430194Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:59.5430695Z 2025-10-10T01:53:59.5431048Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:59.5431394Z 2025-10-10T01:53:59.5431934Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:59.5432456Z 2025-10-10T01:53:59.5432834Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:59.5433189Z 2025-10-10T01:53:59.5433748Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:59.5434276Z 2025-10-10T01:53:59.5434571Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:53:59.5434874Z 2025-10-10T01:53:59.5435521Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:59.5436121Z 2025-10-10T01:53:59.5436527Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:59.5436887Z 2025-10-10T01:53:59.5437139Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5437400Z 2025-10-10T01:53:59.5437965Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:59.5438476Z 2025-10-10T01:53:59.5438786Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:53:59.5439212Z 2025-10-10T01:53:59.5439493Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5439762Z 2025-10-10T01:53:59.5440363Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:59.5440915Z 2025-10-10T01:53:59.5441191Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:53:59.5441476Z 2025-10-10T01:53:59.5441735Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5441999Z 2025-10-10T01:53:59.5442575Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:59.5443107Z 2025-10-10T01:53:59.5443421Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:59.5443728Z 2025-10-10T01:53:59.5444068Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5444349Z 2025-10-10T01:53:59.5444883Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:59.5445364Z 2025-10-10T01:53:59.5445646Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:53:59.5445926Z 2025-10-10T01:53:59.5446231Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5446508Z 2025-10-10T01:53:59.5447036Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:59.5448231Z 2025-10-10T01:53:59.5448549Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:53:59.5448856Z 2025-10-10T01:53:59.5449079Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:53:59.5449323Z 2025-10-10T01:53:59.5449802Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:59.5450246Z 2025-10-10T01:53:59.5450496Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] raise RuntimeError( 2025-10-10T01:53:59.5450749Z 2025-10-10T01:53:59.5451315Z (EngineCore_DP0 pid=8911) ERROR 10-10 01:53:59 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:59.5452004Z (EngineCore_DP0 pid=8911) Process EngineCore_DP0: 2025-10-10T01:53:59.5452433Z (EngineCore_DP0 pid=8911) Traceback (most recent call last): 2025-10-10T01:53:59.5453038Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:53:59.5453548Z (EngineCore_DP0 pid=8911) self.run() 2025-10-10T01:53:59.5454076Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:53:59.5454625Z (EngineCore_DP0 pid=8911) self._target(*self._args, **self._kwargs) 2025-10-10T01:53:59.5455286Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:53:59.5455845Z (EngineCore_DP0 pid=8911) raise e 2025-10-10T01:53:59.5456436Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:53:59.5457084Z (EngineCore_DP0 pid=8911) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:53:59.5457520Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5458136Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:53:59.5458766Z (EngineCore_DP0 pid=8911) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:53:59.5459416Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:53:59.5460030Z (EngineCore_DP0 pid=8911) self.model_executor = executor_class(vllm_config) 2025-10-10T01:53:59.5460522Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5461170Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:53:59.5461749Z (EngineCore_DP0 pid=8911) self._init_executor() 2025-10-10T01:53:59.5462421Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:53:59.5463106Z (EngineCore_DP0 pid=8911) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:53:59.5464005Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:53:59.5464840Z (EngineCore_DP0 pid=8911) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:53:59.5465468Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5466152Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:53:59.5466734Z (EngineCore_DP0 pid=8911) return func(*args, **kwargs) 2025-10-10T01:53:59.5467115Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5467742Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:53:59.5468367Z (EngineCore_DP0 pid=8911) worker_class = resolve_obj_by_qualname( 2025-10-10T01:53:59.5468803Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5469468Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:53:59.5470168Z (EngineCore_DP0 pid=8911) module = importlib.import_module(module_name) 2025-10-10T01:53:59.5470605Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5471182Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:53:59.5471806Z (EngineCore_DP0 pid=8911) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:53:59.5472302Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5472798Z (EngineCore_DP0 pid=8911) File "", line 1387, in _gcd_import 2025-10-10T01:53:59.5473359Z (EngineCore_DP0 pid=8911) File "", line 1360, in _find_and_load 2025-10-10T01:53:59.5474063Z (EngineCore_DP0 pid=8911) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:53:59.5474742Z (EngineCore_DP0 pid=8911) File "", line 935, in _load_unlocked 2025-10-10T01:53:59.5475386Z (EngineCore_DP0 pid=8911) File "", line 999, in exec_module 2025-10-10T01:53:59.5475992Z (EngineCore_DP0 pid=8911) File "", line 488, in _call_with_frames_removed 2025-10-10T01:53:59.5476725Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:53:59.5477401Z (EngineCore_DP0 pid=8911) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:53:59.5478142Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:53:59.5478929Z (EngineCore_DP0 pid=8911) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:53:59.5479777Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:53:59.5480447Z (EngineCore_DP0 pid=8911) class FlashAttentionMetadataBuilder( 2025-10-10T01:53:59.5481228Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:53:59.5482082Z (EngineCore_DP0 pid=8911) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:53:59.5482558Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5483282Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:53:59.5484015Z (EngineCore_DP0 pid=8911) if not is_fa_version_supported(fa_version): 2025-10-10T01:53:59.5484450Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5485186Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:53:59.5485879Z (EngineCore_DP0 pid=8911) return _is_fa2_supported(device)[0] 2025-10-10T01:53:59.5486289Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5486996Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:53:59.5487708Z (EngineCore_DP0 pid=8911) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:53:59.5488194Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5488868Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:53:59.5489501Z (EngineCore_DP0 pid=8911) prop = get_device_properties(device) 2025-10-10T01:53:59.5489913Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:53:59.5490562Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:53:59.5491215Z (EngineCore_DP0 pid=8911) _lazy_init() # will define _get_device_properties 2025-10-10T01:53:59.5491606Z (EngineCore_DP0 pid=8911) ^^^^^^^^^^^^ 2025-10-10T01:53:59.5492184Z (EngineCore_DP0 pid=8911) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:53:59.5492744Z (EngineCore_DP0 pid=8911) raise RuntimeError( 2025-10-10T01:53:59.5493425Z (EngineCore_DP0 pid=8911) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:53:59.9677838Z FAILED 2025-10-10T01:53:59.9807501Z models/test_initialization.py::test_can_initialize_large_subset[Starcoder2ForCausalLM] Fork a new process to run a test 8915 2025-10-10T01:53:59.9818282Z Fork a new process to run a test 0 2025-10-10T01:54:00.0093291Z INFO 10-10 01:54:00 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Starcoder2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'bigcode/starcoder2-3b'} 2025-10-10T01:54:00.1195807Z 2025-10-10T01:54:00.1197100Z config.json: 0% 0.00/700 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:54:07.7793554Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] EngineCore failed to start. 2025-10-10T01:54:07.7794007Z 2025-10-10T01:54:07.7794832Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] Traceback (most recent call last): 2025-10-10T01:54:07.7795403Z 2025-10-10T01:54:07.7796316Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:07.7796963Z 2025-10-10T01:54:07.7797364Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:07.7797746Z 2025-10-10T01:54:07.7798228Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7798576Z 2025-10-10T01:54:07.7799451Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:07.7800332Z 2025-10-10T01:54:07.7800832Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:07.7801419Z 2025-10-10T01:54:07.7802008Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:07.7802452Z 2025-10-10T01:54:07.7802787Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:07.7803238Z 2025-10-10T01:54:07.7803553Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7804003Z 2025-10-10T01:54:07.7804909Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:07.7805762Z 2025-10-10T01:54:07.7806425Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] self._init_executor() 2025-10-10T01:54:07.7806915Z 2025-10-10T01:54:07.7807961Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:07.7808679Z 2025-10-10T01:54:07.7809038Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:07.7809378Z 2025-10-10T01:54:07.7809943Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:07.7810462Z 2025-10-10T01:54:07.7810812Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:07.7811145Z 2025-10-10T01:54:07.7811463Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7811748Z 2025-10-10T01:54:07.7812243Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:07.7819547Z 2025-10-10T01:54:07.7819920Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:54:07.7820232Z 2025-10-10T01:54:07.7820516Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7820781Z 2025-10-10T01:54:07.7821435Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:07.7821958Z 2025-10-10T01:54:07.7822277Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:07.7822568Z 2025-10-10T01:54:07.7822836Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7823111Z 2025-10-10T01:54:07.7823640Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:07.7824178Z 2025-10-10T01:54:07.7824484Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:54:07.7824790Z 2025-10-10T01:54:07.7825066Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7825398Z 2025-10-10T01:54:07.7825855Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:07.7826276Z 2025-10-10T01:54:07.7826657Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:07.7826996Z 2025-10-10T01:54:07.7827282Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7827569Z 2025-10-10T01:54:07.7827908Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:54:07.7828253Z 2025-10-10T01:54:07.7828605Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:54:07.7828957Z 2025-10-10T01:54:07.7829383Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:07.7829747Z 2025-10-10T01:54:07.7830100Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:54:07.7830431Z 2025-10-10T01:54:07.7830804Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:54:07.7831156Z 2025-10-10T01:54:07.7831529Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:07.7831891Z 2025-10-10T01:54:07.7832396Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:07.7832866Z 2025-10-10T01:54:07.7833217Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:07.7833551Z 2025-10-10T01:54:07.7834065Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:07.7834552Z 2025-10-10T01:54:07.7834923Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:07.7835279Z 2025-10-10T01:54:07.7835833Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:07.7836331Z 2025-10-10T01:54:07.7836668Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:54:07.7836964Z 2025-10-10T01:54:07.7837592Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:07.7838156Z 2025-10-10T01:54:07.7838526Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:07.7838913Z 2025-10-10T01:54:07.7839286Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7839557Z 2025-10-10T01:54:07.7840117Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:07.7840685Z 2025-10-10T01:54:07.7840981Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:54:07.7841278Z 2025-10-10T01:54:07.7841547Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7841817Z 2025-10-10T01:54:07.7842416Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:07.7842963Z 2025-10-10T01:54:07.7843251Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:54:07.7843530Z 2025-10-10T01:54:07.7843792Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7844066Z 2025-10-10T01:54:07.7844690Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:07.7845234Z 2025-10-10T01:54:07.7845554Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:07.7845869Z 2025-10-10T01:54:07.7846141Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7846419Z 2025-10-10T01:54:07.7846941Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:07.7847426Z 2025-10-10T01:54:07.7847706Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:54:07.7848008Z 2025-10-10T01:54:07.7848272Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7848538Z 2025-10-10T01:54:07.7849063Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:07.7849541Z 2025-10-10T01:54:07.7849857Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:07.7850158Z 2025-10-10T01:54:07.7850388Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:54:07.7850626Z 2025-10-10T01:54:07.7851104Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:07.7851601Z 2025-10-10T01:54:07.7851852Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] raise RuntimeError( 2025-10-10T01:54:07.7852117Z 2025-10-10T01:54:07.7852684Z (EngineCore_DP0 pid=8973) ERROR 10-10 01:54:07 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:07.7853391Z (EngineCore_DP0 pid=8973) Process EngineCore_DP0: 2025-10-10T01:54:07.7853793Z (EngineCore_DP0 pid=8973) Traceback (most recent call last): 2025-10-10T01:54:07.7854444Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:54:07.7854952Z (EngineCore_DP0 pid=8973) self.run() 2025-10-10T01:54:07.7855478Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:54:07.7856086Z (EngineCore_DP0 pid=8973) self._target(*self._args, **self._kwargs) 2025-10-10T01:54:07.7856763Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:54:07.7857322Z (EngineCore_DP0 pid=8973) raise e 2025-10-10T01:54:07.7857909Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:07.7858544Z (EngineCore_DP0 pid=8973) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:07.7858993Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7859617Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:07.7860261Z (EngineCore_DP0 pid=8973) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:07.7860962Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:07.7861596Z (EngineCore_DP0 pid=8973) self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:07.7862050Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7862712Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:07.7863299Z (EngineCore_DP0 pid=8973) self._init_executor() 2025-10-10T01:54:07.7863965Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:07.7864653Z (EngineCore_DP0 pid=8973) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:07.7865379Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:07.7866088Z (EngineCore_DP0 pid=8973) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:07.7866582Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7867223Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:07.7867804Z (EngineCore_DP0 pid=8973) return func(*args, **kwargs) 2025-10-10T01:54:07.7868188Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7868866Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:07.7869500Z (EngineCore_DP0 pid=8973) worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:07.7869928Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7870597Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:07.7871254Z (EngineCore_DP0 pid=8973) module = importlib.import_module(module_name) 2025-10-10T01:54:07.7871741Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7872320Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:07.7872943Z (EngineCore_DP0 pid=8973) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:07.7873509Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7874010Z (EngineCore_DP0 pid=8973) File "", line 1387, in _gcd_import 2025-10-10T01:54:07.7874565Z (EngineCore_DP0 pid=8973) File "", line 1360, in _find_and_load 2025-10-10T01:54:07.7875154Z (EngineCore_DP0 pid=8973) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:07.7875731Z (EngineCore_DP0 pid=8973) File "", line 935, in _load_unlocked 2025-10-10T01:54:07.7876297Z (EngineCore_DP0 pid=8973) File "", line 999, in exec_module 2025-10-10T01:54:07.7876909Z (EngineCore_DP0 pid=8973) File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:07.7877643Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:07.7878365Z (EngineCore_DP0 pid=8973) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:07.7879171Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:07.7879924Z (EngineCore_DP0 pid=8973) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:07.7880718Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:07.7881400Z (EngineCore_DP0 pid=8973) class FlashAttentionMetadataBuilder( 2025-10-10T01:54:07.7882198Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:07.7883019Z (EngineCore_DP0 pid=8973) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:07.7883502Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7884191Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:07.7884877Z (EngineCore_DP0 pid=8973) if not is_fa_version_supported(fa_version): 2025-10-10T01:54:07.7885319Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7886063Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:07.7886806Z (EngineCore_DP0 pid=8973) return _is_fa2_supported(device)[0] 2025-10-10T01:54:07.7887231Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7887954Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:07.7888673Z (EngineCore_DP0 pid=8973) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:07.7889131Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7889865Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:07.7890496Z (EngineCore_DP0 pid=8973) prop = get_device_properties(device) 2025-10-10T01:54:07.7890919Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:07.7891622Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:07.7892286Z (EngineCore_DP0 pid=8973) _lazy_init() # will define _get_device_properties 2025-10-10T01:54:07.7892675Z (EngineCore_DP0 pid=8973) ^^^^^^^^^^^^ 2025-10-10T01:54:07.7893267Z (EngineCore_DP0 pid=8973) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:07.7893830Z (EngineCore_DP0 pid=8973) raise RuntimeError( 2025-10-10T01:54:07.7894513Z (EngineCore_DP0 pid=8973) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:08.1842115Z FAILED 2025-10-10T01:54:08.1972181Z models/test_initialization.py::test_can_initialize_large_subset[ChatGLMModel] Fork a new process to run a test 8977 2025-10-10T01:54:08.1983069Z Fork a new process to run a test 0 2025-10-10T01:54:08.1987157Z `transformers==4.56.2` installed, but `transformers<=4.48` is required to run this model. 2025-10-10T01:54:08.5018426Z PASSED 2025-10-10T01:54:08.5148103Z models/test_initialization.py::test_can_initialize_large_subset[GraniteForCausalLM] Fork a new process to run a test 8978 2025-10-10T01:54:08.5159577Z Fork a new process to run a test 0 2025-10-10T01:54:08.5435327Z INFO 10-10 01:54:08 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GraniteForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ibm/PowerLM-3b'} 2025-10-10T01:54:08.6812959Z 2025-10-10T01:54:08.6814593Z config.json: 0% 0.00/755 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:54:17.1111329Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] EngineCore failed to start. 2025-10-10T01:54:17.1111805Z 2025-10-10T01:54:17.1112351Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] Traceback (most recent call last): 2025-10-10T01:54:17.1112726Z 2025-10-10T01:54:17.1113579Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:17.1114192Z 2025-10-10T01:54:17.1114586Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:17.1114967Z 2025-10-10T01:54:17.1115313Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1115652Z 2025-10-10T01:54:17.1116519Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:17.1117117Z 2025-10-10T01:54:17.1117590Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:17.1118108Z 2025-10-10T01:54:17.1118868Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:17.1119566Z 2025-10-10T01:54:17.1119978Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:17.1120405Z 2025-10-10T01:54:17.1120696Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1120988Z 2025-10-10T01:54:17.1121623Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:17.1123336Z 2025-10-10T01:54:17.1123721Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] self._init_executor() 2025-10-10T01:54:17.1124199Z 2025-10-10T01:54:17.1124911Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:17.1125616Z 2025-10-10T01:54:17.1126015Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:17.1126375Z 2025-10-10T01:54:17.1127062Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:17.1127596Z 2025-10-10T01:54:17.1128072Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:17.1128426Z 2025-10-10T01:54:17.1128719Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1129002Z 2025-10-10T01:54:17.1129495Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:17.1129959Z 2025-10-10T01:54:17.1130220Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:54:17.1130484Z 2025-10-10T01:54:17.1130752Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1131009Z 2025-10-10T01:54:17.1131531Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:17.1132006Z 2025-10-10T01:54:17.1132298Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:17.1132582Z 2025-10-10T01:54:17.1132842Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1133112Z 2025-10-10T01:54:17.1133631Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:17.1134118Z 2025-10-10T01:54:17.1134418Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:54:17.1134722Z 2025-10-10T01:54:17.1135039Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1135320Z 2025-10-10T01:54:17.1135754Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:17.1136159Z 2025-10-10T01:54:17.1136500Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:17.1136847Z 2025-10-10T01:54:17.1137141Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1137463Z 2025-10-10T01:54:17.1137810Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:54:17.1138138Z 2025-10-10T01:54:17.1138487Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:54:17.1138877Z 2025-10-10T01:54:17.1139248Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:17.1139610Z 2025-10-10T01:54:17.1139958Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:54:17.1140289Z 2025-10-10T01:54:17.1140653Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:54:17.1141005Z 2025-10-10T01:54:17.1141375Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:17.1141730Z 2025-10-10T01:54:17.1142288Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:17.1142751Z 2025-10-10T01:54:17.1143106Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:17.1143444Z 2025-10-10T01:54:17.1143966Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:17.1144442Z 2025-10-10T01:54:17.1144818Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:17.1145175Z 2025-10-10T01:54:17.1145721Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:17.1146229Z 2025-10-10T01:54:17.1146521Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:54:17.1146819Z 2025-10-10T01:54:17.1147451Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:17.1148031Z 2025-10-10T01:54:17.1148403Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:17.1148760Z 2025-10-10T01:54:17.1149012Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1149272Z 2025-10-10T01:54:17.1149896Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:17.1150419Z 2025-10-10T01:54:17.1150721Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:54:17.1151017Z 2025-10-10T01:54:17.1151292Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1151561Z 2025-10-10T01:54:17.1152156Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:17.1152767Z 2025-10-10T01:54:17.1153047Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:54:17.1153331Z 2025-10-10T01:54:17.1153586Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1153929Z 2025-10-10T01:54:17.1154507Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:17.1155039Z 2025-10-10T01:54:17.1155352Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:17.1155659Z 2025-10-10T01:54:17.1155952Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1156234Z 2025-10-10T01:54:17.1156778Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:17.1157270Z 2025-10-10T01:54:17.1157554Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:54:17.1157877Z 2025-10-10T01:54:17.1158151Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1158431Z 2025-10-10T01:54:17.1158967Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:17.1159568Z 2025-10-10T01:54:17.1159889Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:17.1160210Z 2025-10-10T01:54:17.1160434Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:54:17.1160678Z 2025-10-10T01:54:17.1161160Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:17.1161610Z 2025-10-10T01:54:17.1161858Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] raise RuntimeError( 2025-10-10T01:54:17.1162113Z 2025-10-10T01:54:17.1162681Z (EngineCore_DP0 pid=9039) ERROR 10-10 01:54:17 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:17.1163396Z (EngineCore_DP0 pid=9039) Process EngineCore_DP0: 2025-10-10T01:54:17.1163792Z (EngineCore_DP0 pid=9039) Traceback (most recent call last): 2025-10-10T01:54:17.1164392Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:54:17.1164907Z (EngineCore_DP0 pid=9039) self.run() 2025-10-10T01:54:17.1165480Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:54:17.1166049Z (EngineCore_DP0 pid=9039) self._target(*self._args, **self._kwargs) 2025-10-10T01:54:17.1166716Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:54:17.1167273Z (EngineCore_DP0 pid=9039) raise e 2025-10-10T01:54:17.1167859Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:17.1168532Z (EngineCore_DP0 pid=9039) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:17.1168972Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1169590Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:17.1170272Z (EngineCore_DP0 pid=9039) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:17.1170928Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:17.1171547Z (EngineCore_DP0 pid=9039) self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:17.1172000Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1172648Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:17.1173224Z (EngineCore_DP0 pid=9039) self._init_executor() 2025-10-10T01:54:17.1173888Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:17.1174628Z (EngineCore_DP0 pid=9039) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:17.1175365Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:17.1176075Z (EngineCore_DP0 pid=9039) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:17.1176577Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1177214Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:17.1177801Z (EngineCore_DP0 pid=9039) return func(*args, **kwargs) 2025-10-10T01:54:17.1178187Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1178831Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:17.1179462Z (EngineCore_DP0 pid=9039) worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:17.1179878Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1180547Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:17.1181206Z (EngineCore_DP0 pid=9039) module = importlib.import_module(module_name) 2025-10-10T01:54:17.1181648Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1182224Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:17.1182886Z (EngineCore_DP0 pid=9039) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:17.1183382Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1183874Z (EngineCore_DP0 pid=9039) File "", line 1387, in _gcd_import 2025-10-10T01:54:17.1184423Z (EngineCore_DP0 pid=9039) File "", line 1360, in _find_and_load 2025-10-10T01:54:17.1185010Z (EngineCore_DP0 pid=9039) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:17.1185625Z (EngineCore_DP0 pid=9039) File "", line 935, in _load_unlocked 2025-10-10T01:54:17.1186191Z (EngineCore_DP0 pid=9039) File "", line 999, in exec_module 2025-10-10T01:54:17.1186790Z (EngineCore_DP0 pid=9039) File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:17.1187574Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:17.1188257Z (EngineCore_DP0 pid=9039) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:17.1188977Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:17.1189692Z (EngineCore_DP0 pid=9039) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:17.1190469Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:17.1191134Z (EngineCore_DP0 pid=9039) class FlashAttentionMetadataBuilder( 2025-10-10T01:54:17.1191977Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:17.1192795Z (EngineCore_DP0 pid=9039) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:17.1193269Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1193960Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:17.1194641Z (EngineCore_DP0 pid=9039) if not is_fa_version_supported(fa_version): 2025-10-10T01:54:17.1195093Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1195838Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:17.1196808Z (EngineCore_DP0 pid=9039) return _is_fa2_supported(device)[0] 2025-10-10T01:54:17.1197219Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1197931Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:17.1198650Z (EngineCore_DP0 pid=9039) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:17.1199181Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1199850Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:17.1200486Z (EngineCore_DP0 pid=9039) prop = get_device_properties(device) 2025-10-10T01:54:17.1200993Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:17.1201680Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:17.1202347Z (EngineCore_DP0 pid=9039) _lazy_init() # will define _get_device_properties 2025-10-10T01:54:17.1202736Z (EngineCore_DP0 pid=9039) ^^^^^^^^^^^^ 2025-10-10T01:54:17.1203318Z (EngineCore_DP0 pid=9039) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:17.1203965Z (EngineCore_DP0 pid=9039) raise RuntimeError( 2025-10-10T01:54:17.1204644Z (EngineCore_DP0 pid=9039) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:17.5156661Z FAILED 2025-10-10T01:54:17.5286912Z models/test_initialization.py::test_can_initialize_large_subset[Phi3VForCausalLM] Fork a new process to run a test 9043 2025-10-10T01:54:17.5298716Z Fork a new process to run a test 0 2025-10-10T01:54:17.5302992Z `transformers==4.56.2` installed, but `transformers<=4.48` is required to run this model. Reason: Use of deprecated imports which have been removed. 2025-10-10T01:54:17.8310309Z PASSED 2025-10-10T01:54:17.8440805Z models/test_initialization.py::test_can_initialize_large_subset[Tarsier2ForConditionalGeneration] Fork a new process to run a test 9044 2025-10-10T01:54:17.8452090Z Fork a new process to run a test 0 2025-10-10T01:54:17.8725191Z INFO 10-10 01:54:17 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Tarsier2ForConditionalGeneration', exist_overrides={'architectures': ['Tarsier2ForConditionalGeneration']}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'omni-research/Tarsier2-Recap-7b'} 2025-10-10T01:54:17.9915044Z 2025-10-10T01:54:18.2339811Z config.json: 0% 0.00/5.52k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:54:28.0580098Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] EngineCore failed to start. 2025-10-10T01:54:28.0580506Z 2025-10-10T01:54:28.0580907Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] Traceback (most recent call last): 2025-10-10T01:54:28.0581257Z 2025-10-10T01:54:28.0581940Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:28.0582554Z 2025-10-10T01:54:28.0582934Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:28.0583313Z 2025-10-10T01:54:28.0583952Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0584326Z 2025-10-10T01:54:28.0584937Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:28.0585675Z 2025-10-10T01:54:28.0586118Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:28.0586534Z 2025-10-10T01:54:28.0587135Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:28.0587847Z 2025-10-10T01:54:28.0588261Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:28.0588646Z 2025-10-10T01:54:28.0589276Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0589891Z 2025-10-10T01:54:28.0590811Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:28.0591521Z 2025-10-10T01:54:28.0591809Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] self._init_executor() 2025-10-10T01:54:28.0592095Z 2025-10-10T01:54:28.0592733Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:28.0593446Z 2025-10-10T01:54:28.0593802Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:28.0594145Z 2025-10-10T01:54:28.0594830Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:28.0595373Z 2025-10-10T01:54:28.0595725Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:28.0596269Z 2025-10-10T01:54:28.0596640Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0596932Z 2025-10-10T01:54:28.0597492Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:28.0597953Z 2025-10-10T01:54:28.0598223Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:54:28.0598501Z 2025-10-10T01:54:28.0598764Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0599154Z 2025-10-10T01:54:28.0599703Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:28.0600180Z 2025-10-10T01:54:28.0600480Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:28.0600768Z 2025-10-10T01:54:28.0601040Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0601307Z 2025-10-10T01:54:28.0601838Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:28.0602499Z 2025-10-10T01:54:28.0602889Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:54:28.0603213Z 2025-10-10T01:54:28.0603486Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0603775Z 2025-10-10T01:54:28.0604230Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:28.0604658Z 2025-10-10T01:54:28.0605105Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:28.0605455Z 2025-10-10T01:54:28.0605748Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0606091Z 2025-10-10T01:54:28.0606456Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:54:28.0606785Z 2025-10-10T01:54:28.0607149Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:54:28.0607482Z 2025-10-10T01:54:28.0607860Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:28.0608224Z 2025-10-10T01:54:28.0608567Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:54:28.0608914Z 2025-10-10T01:54:28.0609284Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:54:28.0609702Z 2025-10-10T01:54:28.0610235Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:28.0610667Z 2025-10-10T01:54:28.0611294Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:28.0611847Z 2025-10-10T01:54:28.0612219Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:28.0612667Z 2025-10-10T01:54:28.0613231Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:28.0613711Z 2025-10-10T01:54:28.0614104Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:28.0614465Z 2025-10-10T01:54:28.0615043Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:28.0615560Z 2025-10-10T01:54:28.0615858Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:54:28.0616162Z 2025-10-10T01:54:28.0616798Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:28.0617392Z 2025-10-10T01:54:28.0617767Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:28.0618125Z 2025-10-10T01:54:28.0618452Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0618733Z 2025-10-10T01:54:28.0619324Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:28.0619931Z 2025-10-10T01:54:28.0620308Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:54:28.0620654Z 2025-10-10T01:54:28.0620985Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0621348Z 2025-10-10T01:54:28.0621955Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:28.0622545Z 2025-10-10T01:54:28.0622839Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:54:28.0623127Z 2025-10-10T01:54:28.0623387Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0623659Z 2025-10-10T01:54:28.0624244Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:28.0624779Z 2025-10-10T01:54:28.0625096Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:28.0625411Z 2025-10-10T01:54:28.0625681Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0625958Z 2025-10-10T01:54:28.0626562Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:28.0627052Z 2025-10-10T01:54:28.0627343Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:54:28.0627626Z 2025-10-10T01:54:28.0627891Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0628153Z 2025-10-10T01:54:28.0628681Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:28.0629180Z 2025-10-10T01:54:28.0629488Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:28.0629808Z 2025-10-10T01:54:28.0630038Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:54:28.0630296Z 2025-10-10T01:54:28.0630790Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:28.0631245Z 2025-10-10T01:54:28.0631491Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] raise RuntimeError( 2025-10-10T01:54:28.0631746Z 2025-10-10T01:54:28.0632329Z (EngineCore_DP0 pid=9130) ERROR 10-10 01:54:28 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:28.0633043Z (EngineCore_DP0 pid=9130) Process EngineCore_DP0: 2025-10-10T01:54:28.0633435Z (EngineCore_DP0 pid=9130) Traceback (most recent call last): 2025-10-10T01:54:28.0634097Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:54:28.0634637Z (EngineCore_DP0 pid=9130) self.run() 2025-10-10T01:54:28.0635166Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:54:28.0635727Z (EngineCore_DP0 pid=9130) self._target(*self._args, **self._kwargs) 2025-10-10T01:54:28.0636387Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:54:28.0636989Z (EngineCore_DP0 pid=9130) raise e 2025-10-10T01:54:28.0637582Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:28.0638228Z (EngineCore_DP0 pid=9130) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:28.0638731Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0639446Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:28.0640100Z (EngineCore_DP0 pid=9130) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:28.0640759Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:28.0641393Z (EngineCore_DP0 pid=9130) self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:28.0641856Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0642516Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:28.0643101Z (EngineCore_DP0 pid=9130) self._init_executor() 2025-10-10T01:54:28.0643824Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:28.0644522Z (EngineCore_DP0 pid=9130) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:28.0645242Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:28.0645966Z (EngineCore_DP0 pid=9130) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:28.0646458Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0647111Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:28.0647714Z (EngineCore_DP0 pid=9130) return func(*args, **kwargs) 2025-10-10T01:54:28.0648102Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0648744Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:28.0649382Z (EngineCore_DP0 pid=9130) worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:28.0649835Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0650511Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:28.0651183Z (EngineCore_DP0 pid=9130) module = importlib.import_module(module_name) 2025-10-10T01:54:28.0651687Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0652284Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:28.0652914Z (EngineCore_DP0 pid=9130) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:28.0653405Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0653901Z (EngineCore_DP0 pid=9130) File "", line 1387, in _gcd_import 2025-10-10T01:54:28.0654521Z (EngineCore_DP0 pid=9130) File "", line 1360, in _find_and_load 2025-10-10T01:54:28.0655116Z (EngineCore_DP0 pid=9130) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:28.0655714Z (EngineCore_DP0 pid=9130) File "", line 935, in _load_unlocked 2025-10-10T01:54:28.0656348Z (EngineCore_DP0 pid=9130) File "", line 999, in exec_module 2025-10-10T01:54:28.0656953Z (EngineCore_DP0 pid=9130) File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:28.0657692Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:28.0658364Z (EngineCore_DP0 pid=9130) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:28.0659090Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:28.0659836Z (EngineCore_DP0 pid=9130) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:28.0660634Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:28.0661365Z (EngineCore_DP0 pid=9130) class FlashAttentionMetadataBuilder( 2025-10-10T01:54:28.0662180Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:28.0662987Z (EngineCore_DP0 pid=9130) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:28.0663488Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0664184Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:28.0664865Z (EngineCore_DP0 pid=9130) if not is_fa_version_supported(fa_version): 2025-10-10T01:54:28.0665308Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0666062Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:28.0666768Z (EngineCore_DP0 pid=9130) return _is_fa2_supported(device)[0] 2025-10-10T01:54:28.0667184Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0667907Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:28.0668633Z (EngineCore_DP0 pid=9130) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:28.0669085Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0669803Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:28.0670464Z (EngineCore_DP0 pid=9130) prop = get_device_properties(device) 2025-10-10T01:54:28.0670892Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:28.0671548Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:28.0672207Z (EngineCore_DP0 pid=9130) _lazy_init() # will define _get_device_properties 2025-10-10T01:54:28.0672652Z (EngineCore_DP0 pid=9130) ^^^^^^^^^^^^ 2025-10-10T01:54:28.0673235Z (EngineCore_DP0 pid=9130) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:28.0673796Z (EngineCore_DP0 pid=9130) raise RuntimeError( 2025-10-10T01:54:28.0674540Z (EngineCore_DP0 pid=9130) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:28.4841627Z FAILED 2025-10-10T01:54:28.4972154Z models/test_initialization.py::test_can_initialize_large_subset[XLMRobertaForSequenceClassification] Fork a new process to run a test 9134 2025-10-10T01:54:28.4983051Z Fork a new process to run a test 0 2025-10-10T01:54:28.5257473Z INFO 10-10 01:54:28 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='XLMRobertaForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/bge-reranker-v2-m3'} 2025-10-10T01:54:28.6158511Z 2025-10-10T01:54:28.6162295Z config.json: 0% 0.00/795 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:54:30.5512801Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:54:30.5513381Z 2025-10-10T01:54:30.5513798Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:54:30.5514177Z 2025-10-10T01:54:30.5514840Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:30.5515440Z 2025-10-10T01:54:30.5515855Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:30.5516262Z 2025-10-10T01:54:30.5516613Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5516958Z 2025-10-10T01:54:30.5517581Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:30.5518139Z 2025-10-10T01:54:30.5518549Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:30.5518947Z 2025-10-10T01:54:30.5519709Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:30.5520374Z 2025-10-10T01:54:30.5521063Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:30.5521472Z 2025-10-10T01:54:30.5521849Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5522200Z 2025-10-10T01:54:30.5522754Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:30.5523454Z 2025-10-10T01:54:30.5523800Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] self._init_executor() 2025-10-10T01:54:30.5524401Z 2025-10-10T01:54:30.5525310Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:30.5525942Z 2025-10-10T01:54:30.5526643Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:30.5527127Z 2025-10-10T01:54:30.5527717Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:30.5528236Z 2025-10-10T01:54:30.5528585Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:30.5528929Z 2025-10-10T01:54:30.5529219Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5529505Z 2025-10-10T01:54:30.5530001Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:30.5530458Z 2025-10-10T01:54:30.5530824Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:54:30.5531129Z 2025-10-10T01:54:30.5531388Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5531647Z 2025-10-10T01:54:30.5532172Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:30.5532642Z 2025-10-10T01:54:30.5532943Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:30.5533229Z 2025-10-10T01:54:30.5533495Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5533765Z 2025-10-10T01:54:30.5534295Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:30.5534786Z 2025-10-10T01:54:30.5535089Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:54:30.5535396Z 2025-10-10T01:54:30.5535666Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5535941Z 2025-10-10T01:54:30.5536376Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:30.5536804Z 2025-10-10T01:54:30.5537143Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:30.5537467Z 2025-10-10T01:54:30.5537819Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5538102Z 2025-10-10T01:54:30.5538453Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:54:30.5538778Z 2025-10-10T01:54:30.5539128Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:54:30.5539462Z 2025-10-10T01:54:30.5539831Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:30.5540239Z 2025-10-10T01:54:30.5540581Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:54:30.5540924Z 2025-10-10T01:54:30.5541334Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:54:30.5541692Z 2025-10-10T01:54:30.5542069Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:30.5542425Z 2025-10-10T01:54:30.5542925Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:30.5543383Z 2025-10-10T01:54:30.5543739Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:30.5544067Z 2025-10-10T01:54:30.5544594Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:30.5545074Z 2025-10-10T01:54:30.5545512Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:30.5545872Z 2025-10-10T01:54:30.5546425Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:30.5546934Z 2025-10-10T01:54:30.5547223Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:54:30.5547521Z 2025-10-10T01:54:30.5548143Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:30.5548714Z 2025-10-10T01:54:30.5549076Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:30.5549434Z 2025-10-10T01:54:30.5549687Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5549945Z 2025-10-10T01:54:30.5550505Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:30.5551014Z 2025-10-10T01:54:30.5551310Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:54:30.5551601Z 2025-10-10T01:54:30.5551876Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5552151Z 2025-10-10T01:54:30.5552803Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:30.5553359Z 2025-10-10T01:54:30.5553642Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:54:30.5553927Z 2025-10-10T01:54:30.5554184Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5554454Z 2025-10-10T01:54:30.5555039Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:30.5555608Z 2025-10-10T01:54:30.5555923Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:30.5556228Z 2025-10-10T01:54:30.5556544Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5556817Z 2025-10-10T01:54:30.5557356Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:30.5557834Z 2025-10-10T01:54:30.5558120Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:54:30.5558399Z 2025-10-10T01:54:30.5558662Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5558927Z 2025-10-10T01:54:30.5559556Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:30.5560047Z 2025-10-10T01:54:30.5560359Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:30.5560741Z 2025-10-10T01:54:30.5560973Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:54:30.5561217Z 2025-10-10T01:54:30.5561707Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:30.5562156Z 2025-10-10T01:54:30.5562404Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] raise RuntimeError( 2025-10-10T01:54:30.5562661Z 2025-10-10T01:54:30.5563227Z (EngineCore_DP0 pid=9163) ERROR 10-10 01:54:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:30.5563934Z (EngineCore_DP0 pid=9163) Process EngineCore_DP0: 2025-10-10T01:54:30.5564355Z (EngineCore_DP0 pid=9163) Traceback (most recent call last): 2025-10-10T01:54:30.5564957Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:54:30.5565476Z (EngineCore_DP0 pid=9163) self.run() 2025-10-10T01:54:30.5566004Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:54:30.5566562Z (EngineCore_DP0 pid=9163) self._target(*self._args, **self._kwargs) 2025-10-10T01:54:30.5567235Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:54:30.5567777Z (EngineCore_DP0 pid=9163) raise e 2025-10-10T01:54:30.5568407Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:30.5569056Z (EngineCore_DP0 pid=9163) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:30.5569499Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5570115Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:30.5570743Z (EngineCore_DP0 pid=9163) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:30.5571436Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:30.5572046Z (EngineCore_DP0 pid=9163) self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:30.5572517Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5573209Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:30.5573790Z (EngineCore_DP0 pid=9163) self._init_executor() 2025-10-10T01:54:30.5574649Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:30.5575358Z (EngineCore_DP0 pid=9163) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:30.5576102Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:30.5576810Z (EngineCore_DP0 pid=9163) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:30.5577304Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5578648Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:30.5579259Z (EngineCore_DP0 pid=9163) return func(*args, **kwargs) 2025-10-10T01:54:30.5579642Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5580278Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:30.5580904Z (EngineCore_DP0 pid=9163) worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:30.5581322Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5581990Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:30.5582646Z (EngineCore_DP0 pid=9163) module = importlib.import_module(module_name) 2025-10-10T01:54:30.5583092Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5583684Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:30.5584300Z (EngineCore_DP0 pid=9163) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:30.5584792Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5585291Z (EngineCore_DP0 pid=9163) File "", line 1387, in _gcd_import 2025-10-10T01:54:30.5585843Z (EngineCore_DP0 pid=9163) File "", line 1360, in _find_and_load 2025-10-10T01:54:30.5586474Z (EngineCore_DP0 pid=9163) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:30.5587059Z (EngineCore_DP0 pid=9163) File "", line 935, in _load_unlocked 2025-10-10T01:54:30.5587625Z (EngineCore_DP0 pid=9163) File "", line 999, in exec_module 2025-10-10T01:54:30.5588225Z (EngineCore_DP0 pid=9163) File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:30.5588959Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:30.5589694Z (EngineCore_DP0 pid=9163) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:30.5590448Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:30.5591212Z (EngineCore_DP0 pid=9163) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:30.5591992Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:30.5592685Z (EngineCore_DP0 pid=9163) class FlashAttentionMetadataBuilder( 2025-10-10T01:54:30.5593479Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:30.5594295Z (EngineCore_DP0 pid=9163) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:30.5594768Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5595463Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:30.5596498Z (EngineCore_DP0 pid=9163) if not is_fa_version_supported(fa_version): 2025-10-10T01:54:30.5596994Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5597760Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:30.5598461Z (EngineCore_DP0 pid=9163) return _is_fa2_supported(device)[0] 2025-10-10T01:54:30.5598881Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5599668Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:30.5600384Z (EngineCore_DP0 pid=9163) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:30.5600837Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5601505Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:30.5602134Z (EngineCore_DP0 pid=9163) prop = get_device_properties(device) 2025-10-10T01:54:30.5602555Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:30.5603205Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:30.5603870Z (EngineCore_DP0 pid=9163) _lazy_init() # will define _get_device_properties 2025-10-10T01:54:30.5604264Z (EngineCore_DP0 pid=9163) ^^^^^^^^^^^^ 2025-10-10T01:54:30.5604918Z (EngineCore_DP0 pid=9163) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:30.5605499Z (EngineCore_DP0 pid=9163) raise RuntimeError( 2025-10-10T01:54:30.5606183Z (EngineCore_DP0 pid=9163) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:30.9839873Z FAILED 2025-10-10T01:54:30.9969567Z models/test_initialization.py::test_can_initialize_large_subset[GteNewForSequenceClassification] Fork a new process to run a test 9167 2025-10-10T01:54:30.9981256Z Fork a new process to run a test 0 2025-10-10T01:54:31.0262138Z INFO 10-10 01:54:31 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GteNewForSequenceClassification', exist_overrides={'architectures': ['GteNewForSequenceClassification']}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Alibaba-NLP/gte-multilingual-reranker-base'} 2025-10-10T01:54:31.1205573Z 2025-10-10T01:54:31.1207707Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:54:31.1207996Z config.json: 1.49kB [00:00, 9.61MB/s] 2025-10-10T01:54:31.2511293Z INFO 10-10 01:54:31 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:54:38.0114915Z INFO 10-10 01:54:38 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:54:38.0115631Z INFO 10-10 01:54:38 [model.py:551] Resolved architecture: GteNewForSequenceClassification 2025-10-10T01:54:38.0116160Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:54:38.0367412Z INFO 10-10 01:54:38 [model.py:1545] Using max model len 65536 2025-10-10T01:54:38.0369582Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:54:38.0654623Z INFO 10-10 01:54:38 [arg_utils.py:1580] (Disabling) chunked prefill by default 2025-10-10T01:54:38.0655336Z INFO 10-10 01:54:38 [arg_utils.py:1583] (Disabling) prefix caching by default 2025-10-10T01:54:38.0928761Z INFO 10-10 01:54:38 [vllm.py:404] Only "last" pooling supports chunked prefill and prefix caching; disabling both. 2025-10-10T01:54:38.1362564Z 2025-10-10T01:54:38.1364009Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:54:38.1364350Z tokenizer_config.json: 1.34kB [00:00, 12.3MB/s] 2025-10-10T01:54:38.2643632Z 2025-10-10T01:54:38.5816466Z tokenizer.json: 0% 0.00/17.1M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:54:39.6370352Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] EngineCore failed to start. 2025-10-10T01:54:39.6370915Z 2025-10-10T01:54:39.6371237Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] Traceback (most recent call last): 2025-10-10T01:54:39.6371545Z 2025-10-10T01:54:39.6372119Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:39.6372833Z 2025-10-10T01:54:39.6373160Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:39.6373463Z 2025-10-10T01:54:39.6373741Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6374009Z 2025-10-10T01:54:39.6374488Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:39.6374946Z 2025-10-10T01:54:39.6375271Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:39.6375590Z 2025-10-10T01:54:39.6376349Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:39.6376848Z 2025-10-10T01:54:39.6377178Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:39.6377488Z 2025-10-10T01:54:39.6377767Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6378042Z 2025-10-10T01:54:39.6378555Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:39.6379022Z 2025-10-10T01:54:39.6379275Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] self._init_executor() 2025-10-10T01:54:39.6379531Z 2025-10-10T01:54:39.6380290Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:39.6381271Z 2025-10-10T01:54:39.6381866Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:39.6382455Z 2025-10-10T01:54:39.6383397Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:39.6384409Z 2025-10-10T01:54:39.6384912Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:39.6385260Z 2025-10-10T01:54:39.6385552Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6385946Z 2025-10-10T01:54:39.6386462Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:39.6386926Z 2025-10-10T01:54:39.6387190Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:54:39.6387461Z 2025-10-10T01:54:39.6387722Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6387974Z 2025-10-10T01:54:39.6388491Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:39.6388963Z 2025-10-10T01:54:39.6389254Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:39.6389543Z 2025-10-10T01:54:39.6389854Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6390129Z 2025-10-10T01:54:39.6390661Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:39.6391148Z 2025-10-10T01:54:39.6391483Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:54:39.6391788Z 2025-10-10T01:54:39.6392066Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6392338Z 2025-10-10T01:54:39.6392777Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:39.6393193Z 2025-10-10T01:54:39.6393541Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:39.6401514Z 2025-10-10T01:54:39.6402050Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6402370Z 2025-10-10T01:54:39.6402754Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:54:39.6403106Z 2025-10-10T01:54:39.6403471Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:54:39.6403818Z 2025-10-10T01:54:39.6404205Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:39.6404573Z 2025-10-10T01:54:39.6405056Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:54:39.6405424Z 2025-10-10T01:54:39.6405800Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:54:39.6406164Z 2025-10-10T01:54:39.6406538Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:39.6406918Z 2025-10-10T01:54:39.6407509Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:39.6407988Z 2025-10-10T01:54:39.6408360Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:39.6408769Z 2025-10-10T01:54:39.6409318Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:39.6409800Z 2025-10-10T01:54:39.6410180Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:39.6410538Z 2025-10-10T01:54:39.6411094Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:39.6411600Z 2025-10-10T01:54:39.6411902Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:54:39.6412197Z 2025-10-10T01:54:39.6412821Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:39.6413462Z 2025-10-10T01:54:39.6413839Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:39.6414199Z 2025-10-10T01:54:39.6414455Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6414725Z 2025-10-10T01:54:39.6415293Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:39.6415820Z 2025-10-10T01:54:39.6416128Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:54:39.6416424Z 2025-10-10T01:54:39.6416709Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6416982Z 2025-10-10T01:54:39.6417588Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:39.6418134Z 2025-10-10T01:54:39.6418416Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:54:39.6418697Z 2025-10-10T01:54:39.6418954Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6419228Z 2025-10-10T01:54:39.6419803Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:39.6420334Z 2025-10-10T01:54:39.6420697Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:39.6421017Z 2025-10-10T01:54:39.6421291Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6421575Z 2025-10-10T01:54:39.6422093Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:39.6422617Z 2025-10-10T01:54:39.6422905Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:54:39.6423185Z 2025-10-10T01:54:39.6423452Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6423719Z 2025-10-10T01:54:39.6424291Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:39.6424769Z 2025-10-10T01:54:39.6425089Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:39.6425401Z 2025-10-10T01:54:39.6425629Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:54:39.6425876Z 2025-10-10T01:54:39.6426354Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:39.6426809Z 2025-10-10T01:54:39.6427055Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] raise RuntimeError( 2025-10-10T01:54:39.6427326Z 2025-10-10T01:54:39.6427933Z (EngineCore_DP0 pid=9246) ERROR 10-10 01:54:39 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:39.6428849Z (EngineCore_DP0 pid=9246) Process EngineCore_DP0: 2025-10-10T01:54:39.6429247Z (EngineCore_DP0 pid=9246) Traceback (most recent call last): 2025-10-10T01:54:39.6429864Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:54:39.6430393Z (EngineCore_DP0 pid=9246) self.run() 2025-10-10T01:54:39.6430926Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:54:39.6431497Z (EngineCore_DP0 pid=9246) self._target(*self._args, **self._kwargs) 2025-10-10T01:54:39.6432166Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:54:39.6432711Z (EngineCore_DP0 pid=9246) raise e 2025-10-10T01:54:39.6433304Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:39.6433937Z (EngineCore_DP0 pid=9246) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:39.6434380Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6435001Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:39.6435640Z (EngineCore_DP0 pid=9246) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:39.6436299Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:39.6436999Z (EngineCore_DP0 pid=9246) self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:39.6437465Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6438135Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:39.6438724Z (EngineCore_DP0 pid=9246) self._init_executor() 2025-10-10T01:54:39.6439478Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:39.6440216Z (EngineCore_DP0 pid=9246) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:39.6440943Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:39.6441707Z (EngineCore_DP0 pid=9246) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:39.6442208Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6442846Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:39.6443433Z (EngineCore_DP0 pid=9246) return func(*args, **kwargs) 2025-10-10T01:54:39.6443818Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6444460Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:39.6445105Z (EngineCore_DP0 pid=9246) worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:39.6445532Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6446247Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:39.6446910Z (EngineCore_DP0 pid=9246) module = importlib.import_module(module_name) 2025-10-10T01:54:39.6447352Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6447938Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:39.6448558Z (EngineCore_DP0 pid=9246) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:39.6449053Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6449547Z (EngineCore_DP0 pid=9246) File "", line 1387, in _gcd_import 2025-10-10T01:54:39.6450112Z (EngineCore_DP0 pid=9246) File "", line 1360, in _find_and_load 2025-10-10T01:54:39.6450702Z (EngineCore_DP0 pid=9246) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:39.6451275Z (EngineCore_DP0 pid=9246) File "", line 935, in _load_unlocked 2025-10-10T01:54:39.6451845Z (EngineCore_DP0 pid=9246) File "", line 999, in exec_module 2025-10-10T01:54:39.6452448Z (EngineCore_DP0 pid=9246) File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:39.6453182Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:39.6453866Z (EngineCore_DP0 pid=9246) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:39.6454636Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:39.6455385Z (EngineCore_DP0 pid=9246) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:39.6456170Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:39.6456842Z (EngineCore_DP0 pid=9246) class FlashAttentionMetadataBuilder( 2025-10-10T01:54:39.6457680Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:39.6458493Z (EngineCore_DP0 pid=9246) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:39.6459033Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6459729Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:39.6460438Z (EngineCore_DP0 pid=9246) if not is_fa_version_supported(fa_version): 2025-10-10T01:54:39.6460881Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6461627Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:39.6462318Z (EngineCore_DP0 pid=9246) return _is_fa2_supported(device)[0] 2025-10-10T01:54:39.6462735Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6463448Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:39.6464214Z (EngineCore_DP0 pid=9246) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:39.6464677Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6465341Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:39.6465981Z (EngineCore_DP0 pid=9246) prop = get_device_properties(device) 2025-10-10T01:54:39.6466402Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:39.6467062Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:39.6467732Z (EngineCore_DP0 pid=9246) _lazy_init() # will define _get_device_properties 2025-10-10T01:54:39.6468127Z (EngineCore_DP0 pid=9246) ^^^^^^^^^^^^ 2025-10-10T01:54:39.6468714Z (EngineCore_DP0 pid=9246) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:39.6469276Z (EngineCore_DP0 pid=9246) raise RuntimeError( 2025-10-10T01:54:39.6469965Z (EngineCore_DP0 pid=9246) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:40.0703269Z FAILED 2025-10-10T01:54:40.0832865Z models/test_initialization.py::test_can_initialize_large_subset[BaiChuanForCausalLM] Fork a new process to run a test 9250 2025-10-10T01:54:40.0843904Z Fork a new process to run a test 0 2025-10-10T01:54:40.1118564Z INFO 10-10 01:54:40 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BaiChuanForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'baichuan-inc/Baichuan-7B'} 2025-10-10T01:54:40.2386637Z 2025-10-10T01:54:40.2388354Z config.json: 0% 0.00/656 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:54:48.2284653Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] EngineCore failed to start. 2025-10-10T01:54:48.2285109Z 2025-10-10T01:54:48.2285485Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] Traceback (most recent call last): 2025-10-10T01:54:48.2285888Z 2025-10-10T01:54:48.2286569Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:48.2287178Z 2025-10-10T01:54:48.2287824Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:48.2288229Z 2025-10-10T01:54:48.2288617Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2288904Z 2025-10-10T01:54:48.2289397Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:48.2289845Z 2025-10-10T01:54:48.2290178Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:48.2290493Z 2025-10-10T01:54:48.2290972Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:48.2291421Z 2025-10-10T01:54:48.2291740Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:48.2292043Z 2025-10-10T01:54:48.2292317Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2292593Z 2025-10-10T01:54:48.2293093Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:48.2293563Z 2025-10-10T01:54:48.2293810Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] self._init_executor() 2025-10-10T01:54:48.2294076Z 2025-10-10T01:54:48.2294716Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:48.2295240Z 2025-10-10T01:54:48.2295568Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:48.2295882Z 2025-10-10T01:54:48.2296968Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:48.2297827Z 2025-10-10T01:54:48.2298506Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:48.2299058Z 2025-10-10T01:54:48.2299407Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2299788Z 2025-10-10T01:54:48.2300312Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:48.2300896Z 2025-10-10T01:54:48.2301182Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:54:48.2301456Z 2025-10-10T01:54:48.2301708Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2301969Z 2025-10-10T01:54:48.2302487Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:48.2302965Z 2025-10-10T01:54:48.2303261Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:48.2303549Z 2025-10-10T01:54:48.2303824Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2304103Z 2025-10-10T01:54:48.2304696Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:48.2305180Z 2025-10-10T01:54:48.2305508Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:54:48.2305808Z 2025-10-10T01:54:48.2306078Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2306356Z 2025-10-10T01:54:48.2306793Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:48.2307205Z 2025-10-10T01:54:48.2307552Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:48.2307890Z 2025-10-10T01:54:48.2308177Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2308458Z 2025-10-10T01:54:48.2308797Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:54:48.2309124Z 2025-10-10T01:54:48.2309478Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:54:48.2309813Z 2025-10-10T01:54:48.2310189Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:48.2310543Z 2025-10-10T01:54:48.2310956Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:54:48.2311296Z 2025-10-10T01:54:48.2311670Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:54:48.2312025Z 2025-10-10T01:54:48.2312400Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:48.2312773Z 2025-10-10T01:54:48.2313266Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:48.2313846Z 2025-10-10T01:54:48.2314213Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:48.2314552Z 2025-10-10T01:54:48.2315085Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:48.2315602Z 2025-10-10T01:54:48.2315993Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:48.2316352Z 2025-10-10T01:54:48.2316903Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:48.2317404Z 2025-10-10T01:54:48.2317703Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:54:48.2317998Z 2025-10-10T01:54:48.2318629Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:48.2319342Z 2025-10-10T01:54:48.2319761Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:48.2320119Z 2025-10-10T01:54:48.2320373Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2320638Z 2025-10-10T01:54:48.2321191Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:48.2321727Z 2025-10-10T01:54:48.2322026Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:54:48.2322327Z 2025-10-10T01:54:48.2322593Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2322874Z 2025-10-10T01:54:48.2323482Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:48.2324039Z 2025-10-10T01:54:48.2324321Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:54:48.2324601Z 2025-10-10T01:54:48.2324858Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2325119Z 2025-10-10T01:54:48.2325692Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:48.2326221Z 2025-10-10T01:54:48.2326583Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:48.2326906Z 2025-10-10T01:54:48.2327188Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2327465Z 2025-10-10T01:54:48.2327992Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:48.2328478Z 2025-10-10T01:54:48.2328763Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:54:48.2329085Z 2025-10-10T01:54:48.2329362Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2329626Z 2025-10-10T01:54:48.2330145Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:48.2330672Z 2025-10-10T01:54:48.2330990Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:48.2331291Z 2025-10-10T01:54:48.2331514Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:54:48.2331754Z 2025-10-10T01:54:48.2332233Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:48.2332683Z 2025-10-10T01:54:48.2332924Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] raise RuntimeError( 2025-10-10T01:54:48.2333181Z 2025-10-10T01:54:48.2333745Z (EngineCore_DP0 pid=9328) ERROR 10-10 01:54:48 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:48.2334488Z (EngineCore_DP0 pid=9328) Process EngineCore_DP0: 2025-10-10T01:54:48.2334910Z (EngineCore_DP0 pid=9328) Traceback (most recent call last): 2025-10-10T01:54:48.2335519Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:54:48.2336033Z (EngineCore_DP0 pid=9328) self.run() 2025-10-10T01:54:48.2336563Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:54:48.2337132Z (EngineCore_DP0 pid=9328) self._target(*self._args, **self._kwargs) 2025-10-10T01:54:48.2337797Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:54:48.2338344Z (EngineCore_DP0 pid=9328) raise e 2025-10-10T01:54:48.2338952Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:54:48.2339574Z (EngineCore_DP0 pid=9328) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:54:48.2340020Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2340640Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:54:48.2341294Z (EngineCore_DP0 pid=9328) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:54:48.2341951Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:54:48.2342557Z (EngineCore_DP0 pid=9328) self.model_executor = executor_class(vllm_config) 2025-10-10T01:54:48.2343059Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2343723Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:54:48.2344315Z (EngineCore_DP0 pid=9328) self._init_executor() 2025-10-10T01:54:48.2344972Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:54:48.2345705Z (EngineCore_DP0 pid=9328) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:54:48.2346426Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:54:48.2347134Z (EngineCore_DP0 pid=9328) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:54:48.2347692Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2348332Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:54:48.2348914Z (EngineCore_DP0 pid=9328) return func(*args, **kwargs) 2025-10-10T01:54:48.2349289Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2349923Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:54:48.2350549Z (EngineCore_DP0 pid=9328) worker_class = resolve_obj_by_qualname( 2025-10-10T01:54:48.2350967Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2351625Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:54:48.2352321Z (EngineCore_DP0 pid=9328) module = importlib.import_module(module_name) 2025-10-10T01:54:48.2352769Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2353348Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:54:48.2353967Z (EngineCore_DP0 pid=9328) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:54:48.2354460Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2354951Z (EngineCore_DP0 pid=9328) File "", line 1387, in _gcd_import 2025-10-10T01:54:48.2355505Z (EngineCore_DP0 pid=9328) File "", line 1360, in _find_and_load 2025-10-10T01:54:48.2356106Z (EngineCore_DP0 pid=9328) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:48.2356680Z (EngineCore_DP0 pid=9328) File "", line 935, in _load_unlocked 2025-10-10T01:54:48.2357246Z (EngineCore_DP0 pid=9328) File "", line 999, in exec_module 2025-10-10T01:54:48.2357837Z (EngineCore_DP0 pid=9328) File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:48.2358567Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:54:48.2359344Z (EngineCore_DP0 pid=9328) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:54:48.2360144Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:54:48.2360898Z (EngineCore_DP0 pid=9328) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:54:48.2361676Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:54:48.2362345Z (EngineCore_DP0 pid=9328) class FlashAttentionMetadataBuilder( 2025-10-10T01:54:48.2363129Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:54:48.2363998Z (EngineCore_DP0 pid=9328) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:54:48.2364478Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2365240Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:54:48.2365919Z (EngineCore_DP0 pid=9328) if not is_fa_version_supported(fa_version): 2025-10-10T01:54:48.2366355Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2367094Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:54:48.2367787Z (EngineCore_DP0 pid=9328) return _is_fa2_supported(device)[0] 2025-10-10T01:54:48.2368200Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2368903Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:54:48.2369622Z (EngineCore_DP0 pid=9328) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:54:48.2370131Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2370810Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:48.2371445Z (EngineCore_DP0 pid=9328) prop = get_device_properties(device) 2025-10-10T01:54:48.2371864Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:48.2372523Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:48.2373185Z (EngineCore_DP0 pid=9328) _lazy_init() # will define _get_device_properties 2025-10-10T01:54:48.2373580Z (EngineCore_DP0 pid=9328) ^^^^^^^^^^^^ 2025-10-10T01:54:48.2374179Z (EngineCore_DP0 pid=9328) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:48.2374732Z (EngineCore_DP0 pid=9328) raise RuntimeError( 2025-10-10T01:54:48.2375429Z (EngineCore_DP0 pid=9328) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:48.6328532Z FAILED 2025-10-10T01:54:48.6459919Z models/test_initialization.py::test_can_initialize_large_subset[Mistral3ForConditionalGeneration] Fork a new process to run a test 9332 2025-10-10T01:54:48.6470694Z Fork a new process to run a test 0 2025-10-10T01:54:48.6748207Z INFO 10-10 01:54:48 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Mistral3ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mistralai/Mistral-Small-3.1-24B-Instruct-2503'} 2025-10-10T01:54:48.8970662Z 2025-10-10T01:54:48.8971533Z config.json: 0% 0.00/1.18k [00:00", line 1387, in _gcd_import 2025-10-10T01:54:57.5631968Z 2025-10-10T01:54:57.5632178Z ERROR 10-10 01:54:57 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T01:54:57.5632458Z 2025-10-10T01:54:57.5632793Z ERROR 10-10 01:54:57 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:54:57.5633106Z 2025-10-10T01:54:57.5633314Z ERROR 10-10 01:54:57 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T01:54:57.5633588Z 2025-10-10T01:54:57.5633802Z ERROR 10-10 01:54:57 [registry.py:542] File "", line 999, in exec_module 2025-10-10T01:54:57.5634108Z 2025-10-10T01:54:57.5634336Z ERROR 10-10 01:54:57 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T01:54:57.5634743Z 2025-10-10T01:54:57.5635118Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/mistral3.py", line 39, in 2025-10-10T01:54:57.5635551Z 2025-10-10T01:54:57.5635766Z ERROR 10-10 01:54:57 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T01:54:57.5636130Z 2025-10-10T01:54:57.5636500Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T01:54:57.5636930Z 2025-10-10T01:54:57.5637081Z ERROR 10-10 01:54:57 [registry.py:542] from xformers import ops as xops 2025-10-10T01:54:57.5637294Z 2025-10-10T01:54:57.5637608Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T01:54:57.5638074Z 2025-10-10T01:54:57.5638238Z ERROR 10-10 01:54:57 [registry.py:542] from .fmha import ( 2025-10-10T01:54:57.5638510Z 2025-10-10T01:54:57.5639116Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T01:54:57.5639688Z 2025-10-10T01:54:57.5639928Z ERROR 10-10 01:54:57 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T01:54:57.5640232Z 2025-10-10T01:54:57.5640654Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T01:54:57.5641085Z 2025-10-10T01:54:57.5641247Z ERROR 10-10 01:54:57 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T01:54:57.5641477Z 2025-10-10T01:54:57.5641614Z ERROR 10-10 01:54:57 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:57.5641822Z 2025-10-10T01:54:57.5642140Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T01:54:57.5642529Z 2025-10-10T01:54:57.5642634Z ERROR 10-10 01:54:57 [registry.py:542] value = func() 2025-10-10T01:54:57.5642813Z 2025-10-10T01:54:57.5642913Z ERROR 10-10 01:54:57 [registry.py:542] ^^^^^^ 2025-10-10T01:54:57.5643102Z 2025-10-10T01:54:57.5643442Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T01:54:57.5643849Z 2025-10-10T01:54:57.5644031Z ERROR 10-10 01:54:57 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T01:54:57.5644284Z 2025-10-10T01:54:57.5644423Z ERROR 10-10 01:54:57 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:57.5644634Z 2025-10-10T01:54:57.5644988Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:54:57.5645409Z 2025-10-10T01:54:57.5645555Z ERROR 10-10 01:54:57 [registry.py:542] prop = get_device_properties(device) 2025-10-10T01:54:57.5645781Z 2025-10-10T01:54:57.5645909Z ERROR 10-10 01:54:57 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:54:57.5646115Z 2025-10-10T01:54:57.5646507Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:54:57.5646942Z 2025-10-10T01:54:57.5647113Z ERROR 10-10 01:54:57 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T01:54:57.5647358Z 2025-10-10T01:54:57.5647481Z ERROR 10-10 01:54:57 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T01:54:57.5647651Z 2025-10-10T01:54:57.5647967Z ERROR 10-10 01:54:57 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:54:57.5648401Z 2025-10-10T01:54:57.5648518Z ERROR 10-10 01:54:57 [registry.py:542] raise RuntimeError( 2025-10-10T01:54:57.5648713Z 2025-10-10T01:54:57.5649110Z ERROR 10-10 01:54:57 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:54:57.9539925Z FAILED 2025-10-10T01:54:57.9669477Z models/test_initialization.py::test_can_initialize_large_subset[TransformersMoEForMultimodalLM] Fork a new process to run a test 9408 2025-10-10T01:54:57.9681747Z Fork a new process to run a test 0 2025-10-10T01:54:57.9685481Z `transformers==4.56.2` installed, but `transformers>=4.57.0.dev0` is required to run this model. 2025-10-10T01:54:58.2680919Z PASSED 2025-10-10T01:54:58.2810812Z models/test_initialization.py::test_can_initialize_large_subset[LongCatFlashMTPModel] Fork a new process to run a test 9409 2025-10-10T01:54:58.2823305Z Fork a new process to run a test 0 2025-10-10T01:54:58.3099618Z INFO 10-10 01:54:58 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LongCatFlashMTPModel', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'meituan-longcat/LongCat-Flash-Chat', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'meituan-longcat/LongCat-Flash-Chat'} 2025-10-10T01:54:58.8265958Z 2025-10-10T01:54:58.8267691Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:54:58.8268022Z config.json: 1.00kB [00:00, 5.00MB/s] 2025-10-10T01:54:58.9110767Z 2025-10-10T01:54:58.9112636Z configuration_longcat_flash.py: 0.00B [00:00, ?B/s] 2025-10-10T01:54:58.9113061Z configuration_longcat_flash.py: 9.69kB [00:00, 57.2MB/s] 2025-10-10T01:54:58.9206095Z A new version of the following files was downloaded from https://huggingface.co/meituan-longcat/LongCat-Flash-Chat: 2025-10-10T01:54:58.9206705Z - configuration_longcat_flash.py 2025-10-10T01:54:58.9207330Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:55:05.6816841Z INFO 10-10 01:55:05 [model.py:551] Resolved architecture: LongcatFlashForCausalLM 2025-10-10T01:55:05.6817558Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:55:05.7855712Z 2025-10-10T01:55:05.7974762Z model.safetensors.index.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:05.7975182Z model.safetensors.index.json: 4.14MB [00:00, 349MB/s] 2025-10-10T01:55:05.8421744Z 2025-10-10T01:55:06.0749728Z Parse safetensors files: 0% 0/75 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:55:15.0927644Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:15.0928345Z 2025-10-10T01:55:15.0928761Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:15.0929120Z 2025-10-10T01:55:15.0929789Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:15.0930414Z 2025-10-10T01:55:15.0930813Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:15.0931318Z 2025-10-10T01:55:15.0931691Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0932064Z 2025-10-10T01:55:15.0932822Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:15.0933609Z 2025-10-10T01:55:15.0933988Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:15.0934312Z 2025-10-10T01:55:15.0934802Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:15.0935250Z 2025-10-10T01:55:15.0935571Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:15.0935883Z 2025-10-10T01:55:15.0936169Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0936444Z 2025-10-10T01:55:15.0937097Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:15.0937788Z 2025-10-10T01:55:15.0938210Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] self._init_executor() 2025-10-10T01:55:15.0938713Z 2025-10-10T01:55:15.0939611Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:15.0940494Z 2025-10-10T01:55:15.0941076Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:15.0941634Z 2025-10-10T01:55:15.0942293Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:15.0942811Z 2025-10-10T01:55:15.0943177Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:15.0943529Z 2025-10-10T01:55:15.0943845Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0944133Z 2025-10-10T01:55:15.0944634Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:15.0945100Z 2025-10-10T01:55:15.0945368Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:15.0945640Z 2025-10-10T01:55:15.0945888Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0946188Z 2025-10-10T01:55:15.0946793Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:15.0947282Z 2025-10-10T01:55:15.0947580Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:15.0947871Z 2025-10-10T01:55:15.0948141Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0948408Z 2025-10-10T01:55:15.0948933Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:15.0949463Z 2025-10-10T01:55:15.0949766Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:15.0950085Z 2025-10-10T01:55:15.0950362Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0950685Z 2025-10-10T01:55:15.0951122Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:15.0951534Z 2025-10-10T01:55:15.0951872Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:15.0952197Z 2025-10-10T01:55:15.0952486Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0952774Z 2025-10-10T01:55:15.0953128Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:15.0953456Z 2025-10-10T01:55:15.0953805Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:15.0954139Z 2025-10-10T01:55:15.0954551Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:15.0954916Z 2025-10-10T01:55:15.0955255Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:15.0955592Z 2025-10-10T01:55:15.0955949Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:15.0956306Z 2025-10-10T01:55:15.0956676Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:15.0957040Z 2025-10-10T01:55:15.0957533Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:15.0957991Z 2025-10-10T01:55:15.0958360Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:15.0958690Z 2025-10-10T01:55:15.0959357Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:15.0959830Z 2025-10-10T01:55:15.0960207Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:15.0960573Z 2025-10-10T01:55:15.0961134Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:15.0961642Z 2025-10-10T01:55:15.0961979Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:15.0962284Z 2025-10-10T01:55:15.0962917Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:15.0963486Z 2025-10-10T01:55:15.0963847Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:15.0964253Z 2025-10-10T01:55:15.0964508Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0964773Z 2025-10-10T01:55:15.0965331Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:15.0965914Z 2025-10-10T01:55:15.0966221Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:15.0966513Z 2025-10-10T01:55:15.0966792Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0967063Z 2025-10-10T01:55:15.0967677Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:15.0968227Z 2025-10-10T01:55:15.0968507Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:15.0968795Z 2025-10-10T01:55:15.0969052Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0969328Z 2025-10-10T01:55:15.0969940Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:15.0970467Z 2025-10-10T01:55:15.0970785Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:15.0971097Z 2025-10-10T01:55:15.0971366Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0971639Z 2025-10-10T01:55:15.0972163Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:15.0972641Z 2025-10-10T01:55:15.0972935Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:15.0973229Z 2025-10-10T01:55:15.0973497Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0973762Z 2025-10-10T01:55:15.0974291Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:15.0974772Z 2025-10-10T01:55:15.0975079Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:15.0975386Z 2025-10-10T01:55:15.0975611Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:15.0975854Z 2025-10-10T01:55:15.0976373Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:15.0976833Z 2025-10-10T01:55:15.0977076Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] raise RuntimeError( 2025-10-10T01:55:15.0977328Z 2025-10-10T01:55:15.0977896Z (EngineCore_DP0 pid=9557) ERROR 10-10 01:55:15 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:15.0978591Z (EngineCore_DP0 pid=9557) Process EngineCore_DP0: 2025-10-10T01:55:15.0978985Z (EngineCore_DP0 pid=9557) Traceback (most recent call last): 2025-10-10T01:55:15.0979654Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:15.0980164Z (EngineCore_DP0 pid=9557) self.run() 2025-10-10T01:55:15.0980689Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:15.0981295Z (EngineCore_DP0 pid=9557) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:15.0981954Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:15.0982501Z (EngineCore_DP0 pid=9557) raise e 2025-10-10T01:55:15.0983093Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:15.0983736Z (EngineCore_DP0 pid=9557) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:15.0984180Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0984791Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:15.0985434Z (EngineCore_DP0 pid=9557) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:15.0986130Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:15.0986766Z (EngineCore_DP0 pid=9557) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:15.0987216Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0987857Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:15.0988440Z (EngineCore_DP0 pid=9557) self._init_executor() 2025-10-10T01:55:15.0989102Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:15.0989787Z (EngineCore_DP0 pid=9557) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:15.0990511Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:15.0991217Z (EngineCore_DP0 pid=9557) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:15.0991707Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0992351Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:15.0992936Z (EngineCore_DP0 pid=9557) return func(*args, **kwargs) 2025-10-10T01:55:15.0993318Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0993999Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:15.0994627Z (EngineCore_DP0 pid=9557) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:15.0995057Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0995724Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:15.0996602Z (EngineCore_DP0 pid=9557) module = importlib.import_module(module_name) 2025-10-10T01:55:15.0997141Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0997736Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:15.0998366Z (EngineCore_DP0 pid=9557) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:15.0998944Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.0999521Z (EngineCore_DP0 pid=9557) File "", line 1387, in _gcd_import 2025-10-10T01:55:15.1000079Z (EngineCore_DP0 pid=9557) File "", line 1360, in _find_and_load 2025-10-10T01:55:15.1000659Z (EngineCore_DP0 pid=9557) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:15.1001247Z (EngineCore_DP0 pid=9557) File "", line 935, in _load_unlocked 2025-10-10T01:55:15.1001822Z (EngineCore_DP0 pid=9557) File "", line 999, in exec_module 2025-10-10T01:55:15.1002424Z (EngineCore_DP0 pid=9557) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:15.1003220Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:15.1003896Z (EngineCore_DP0 pid=9557) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:15.1004614Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:15.1005329Z (EngineCore_DP0 pid=9557) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:15.1006108Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:15.1006773Z (EngineCore_DP0 pid=9557) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:15.1007564Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:15.1008374Z (EngineCore_DP0 pid=9557) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:15.1008851Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.1009537Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:15.1010248Z (EngineCore_DP0 pid=9557) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:15.1010695Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.1011436Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:15.1012201Z (EngineCore_DP0 pid=9557) return _is_fa2_supported(device)[0] 2025-10-10T01:55:15.1012617Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.1013325Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:15.1014038Z (EngineCore_DP0 pid=9557) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:15.1014494Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.1015227Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:15.1015853Z (EngineCore_DP0 pid=9557) prop = get_device_properties(device) 2025-10-10T01:55:15.1016272Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:15.1016989Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:15.1017647Z (EngineCore_DP0 pid=9557) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:15.1018039Z (EngineCore_DP0 pid=9557) ^^^^^^^^^^^^ 2025-10-10T01:55:15.1018612Z (EngineCore_DP0 pid=9557) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:15.1019169Z (EngineCore_DP0 pid=9557) raise RuntimeError( 2025-10-10T01:55:15.1019854Z (EngineCore_DP0 pid=9557) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:15.5124631Z FAILED 2025-10-10T01:55:15.5259328Z models/test_initialization.py::test_can_initialize_large_subset[InternVLForConditionalGeneration] Fork a new process to run a test 9561 2025-10-10T01:55:15.5270822Z Fork a new process to run a test 0 2025-10-10T01:55:15.5548983Z INFO 10-10 01:55:15 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternVLForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'OpenGVLab/InternVL3-1B-hf'} 2025-10-10T01:55:15.6918702Z 2025-10-10T01:55:15.6921092Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:15.6921409Z config.json: 1.96kB [00:00, 9.81MB/s] 2025-10-10T01:55:15.8207037Z 2025-10-10T01:55:15.8207557Z preprocessor_config.json: 0% 0.00/666 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:55:17.8786967Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:17.8787577Z 2025-10-10T01:55:17.8788037Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:17.8788489Z 2025-10-10T01:55:17.8789343Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:17.8790113Z 2025-10-10T01:55:17.8790527Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:17.8791068Z 2025-10-10T01:55:17.8791636Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8792207Z 2025-10-10T01:55:17.8793011Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:17.8793582Z 2025-10-10T01:55:17.8794180Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:17.8794636Z 2025-10-10T01:55:17.8795386Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:17.8796443Z 2025-10-10T01:55:17.8797039Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:17.8797634Z 2025-10-10T01:55:17.8798099Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8798766Z 2025-10-10T01:55:17.8799817Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:17.8800377Z 2025-10-10T01:55:17.8800649Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] self._init_executor() 2025-10-10T01:55:17.8800918Z 2025-10-10T01:55:17.8801480Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:17.8802002Z 2025-10-10T01:55:17.8802324Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:17.8802660Z 2025-10-10T01:55:17.8803338Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:17.8803855Z 2025-10-10T01:55:17.8804221Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:17.8804565Z 2025-10-10T01:55:17.8804868Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8805148Z 2025-10-10T01:55:17.8805658Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:17.8806113Z 2025-10-10T01:55:17.8806374Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:17.8806646Z 2025-10-10T01:55:17.8806894Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8807165Z 2025-10-10T01:55:17.8807682Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:17.8808158Z 2025-10-10T01:55:17.8808442Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:17.8808726Z 2025-10-10T01:55:17.8808996Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8809274Z 2025-10-10T01:55:17.8809809Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:17.8810291Z 2025-10-10T01:55:17.8810672Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:17.8810982Z 2025-10-10T01:55:17.8811259Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8811538Z 2025-10-10T01:55:17.8811978Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:17.8812392Z 2025-10-10T01:55:17.8812727Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:17.8813120Z 2025-10-10T01:55:17.8813406Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8813689Z 2025-10-10T01:55:17.8814029Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:17.8814395Z 2025-10-10T01:55:17.8814751Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:17.8815096Z 2025-10-10T01:55:17.8815483Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:17.8815839Z 2025-10-10T01:55:17.8816187Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:17.8816517Z 2025-10-10T01:55:17.8816873Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:17.8817229Z 2025-10-10T01:55:17.8817598Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:17.8817964Z 2025-10-10T01:55:17.8818508Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:17.8818979Z 2025-10-10T01:55:17.8819328Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:17.8819661Z 2025-10-10T01:55:17.8820172Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:17.8820660Z 2025-10-10T01:55:17.8821037Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:17.8821396Z 2025-10-10T01:55:17.8821966Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:17.8822482Z 2025-10-10T01:55:17.8822778Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:17.8823071Z 2025-10-10T01:55:17.8823703Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:17.8824282Z 2025-10-10T01:55:17.8824654Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:17.8825019Z 2025-10-10T01:55:17.8825319Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8825592Z 2025-10-10T01:55:17.8826153Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:17.8826672Z 2025-10-10T01:55:17.8826964Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:17.8827260Z 2025-10-10T01:55:17.8827539Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8827855Z 2025-10-10T01:55:17.8828460Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:17.8829007Z 2025-10-10T01:55:17.8829379Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:17.8829664Z 2025-10-10T01:55:17.8829928Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8830192Z 2025-10-10T01:55:17.8830777Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:17.8831308Z 2025-10-10T01:55:17.8831621Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:17.8831933Z 2025-10-10T01:55:17.8832204Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8832491Z 2025-10-10T01:55:17.8833016Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:17.8833542Z 2025-10-10T01:55:17.8833824Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:17.8834107Z 2025-10-10T01:55:17.8834372Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8834637Z 2025-10-10T01:55:17.8835159Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:17.8835637Z 2025-10-10T01:55:17.8835949Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:17.8836261Z 2025-10-10T01:55:17.8836483Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:17.8836730Z 2025-10-10T01:55:17.8837207Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:17.8837655Z 2025-10-10T01:55:17.8837896Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] raise RuntimeError( 2025-10-10T01:55:17.8838155Z 2025-10-10T01:55:17.8838714Z (EngineCore_DP0 pid=9589) ERROR 10-10 01:55:17 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:17.8839535Z (EngineCore_DP0 pid=9589) Process EngineCore_DP0: 2025-10-10T01:55:17.8839927Z (EngineCore_DP0 pid=9589) Traceback (most recent call last): 2025-10-10T01:55:17.8840573Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:17.8841095Z (EngineCore_DP0 pid=9589) self.run() 2025-10-10T01:55:17.8841618Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:17.8842174Z (EngineCore_DP0 pid=9589) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:17.8842837Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:17.8843374Z (EngineCore_DP0 pid=9589) raise e 2025-10-10T01:55:17.8844002Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:17.8844629Z (EngineCore_DP0 pid=9589) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:17.8845142Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8845762Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:17.8846400Z (EngineCore_DP0 pid=9589) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:17.8847050Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:17.8847662Z (EngineCore_DP0 pid=9589) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:17.8848113Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8848772Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:17.8849354Z (EngineCore_DP0 pid=9589) self._init_executor() 2025-10-10T01:55:17.8850052Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:17.8850739Z (EngineCore_DP0 pid=9589) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:17.8851464Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:17.8852174Z (EngineCore_DP0 pid=9589) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:17.8852670Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8853301Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:17.8853890Z (EngineCore_DP0 pid=9589) return func(*args, **kwargs) 2025-10-10T01:55:17.8854287Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8854923Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:17.8855551Z (EngineCore_DP0 pid=9589) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:17.8855966Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8856630Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:17.8857283Z (EngineCore_DP0 pid=9589) module = importlib.import_module(module_name) 2025-10-10T01:55:17.8857721Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8858359Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:17.8858985Z (EngineCore_DP0 pid=9589) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:17.8859468Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8859964Z (EngineCore_DP0 pid=9589) File "", line 1387, in _gcd_import 2025-10-10T01:55:17.8860512Z (EngineCore_DP0 pid=9589) File "", line 1360, in _find_and_load 2025-10-10T01:55:17.8861135Z (EngineCore_DP0 pid=9589) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:17.8861709Z (EngineCore_DP0 pid=9589) File "", line 935, in _load_unlocked 2025-10-10T01:55:17.8862321Z (EngineCore_DP0 pid=9589) File "", line 999, in exec_module 2025-10-10T01:55:17.8862923Z (EngineCore_DP0 pid=9589) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:17.8863658Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:17.8864334Z (EngineCore_DP0 pid=9589) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:17.8865057Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:17.8865780Z (EngineCore_DP0 pid=9589) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:17.8866550Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:17.8867264Z (EngineCore_DP0 pid=9589) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:17.8868053Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:17.8868865Z (EngineCore_DP0 pid=9589) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:17.8869339Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8870022Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:17.8870720Z (EngineCore_DP0 pid=9589) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:17.8871156Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8871902Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:17.8872610Z (EngineCore_DP0 pid=9589) return _is_fa2_supported(device)[0] 2025-10-10T01:55:17.8873018Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8880465Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:17.8881337Z (EngineCore_DP0 pid=9589) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:17.8881823Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8882604Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:17.8883279Z (EngineCore_DP0 pid=9589) prop = get_device_properties(device) 2025-10-10T01:55:17.8883710Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:17.8884409Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:17.8885118Z (EngineCore_DP0 pid=9589) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:17.8885522Z (EngineCore_DP0 pid=9589) ^^^^^^^^^^^^ 2025-10-10T01:55:17.8886165Z (EngineCore_DP0 pid=9589) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:17.8886735Z (EngineCore_DP0 pid=9589) raise RuntimeError( 2025-10-10T01:55:17.8887426Z (EngineCore_DP0 pid=9589) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:18.2983813Z FAILED 2025-10-10T01:55:18.3112507Z models/test_initialization.py::test_can_initialize_large_subset[MptForCausalLM] Fork a new process to run a test 9593 2025-10-10T01:55:18.3124702Z Fork a new process to run a test 0 2025-10-10T01:55:18.3126141Z Model is not available online 2025-10-10T01:55:18.6154116Z PASSED 2025-10-10T01:55:18.6283610Z models/test_initialization.py::test_can_initialize_large_subset[Phi3ForCausalLM] Fork a new process to run a test 9594 2025-10-10T01:55:18.6295617Z Fork a new process to run a test 0 2025-10-10T01:55:18.6570937Z INFO 10-10 01:55:18 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Phi3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'microsoft/Phi-3-mini-4k-instruct'} 2025-10-10T01:55:18.7438277Z 2025-10-10T01:55:18.7439693Z config.json: 0% 0.00/967 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:55:26.5112152Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:26.5112736Z 2025-10-10T01:55:26.5113298Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:26.5113614Z 2025-10-10T01:55:26.5114276Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:26.5114783Z 2025-10-10T01:55:26.5115107Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:26.5115418Z 2025-10-10T01:55:26.5115697Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5115981Z 2025-10-10T01:55:26.5116498Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:26.5116946Z 2025-10-10T01:55:26.5117279Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:26.5117595Z 2025-10-10T01:55:26.5118072Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:26.5118527Z 2025-10-10T01:55:26.5119069Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:26.5119460Z 2025-10-10T01:55:26.5119835Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5120250Z 2025-10-10T01:55:26.5120773Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:26.5121251Z 2025-10-10T01:55:26.5121518Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] self._init_executor() 2025-10-10T01:55:26.5121785Z 2025-10-10T01:55:26.5122322Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:26.5123045Z 2025-10-10T01:55:26.5123411Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:26.5123923Z 2025-10-10T01:55:26.5124525Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:26.5125280Z 2025-10-10T01:55:26.5125780Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:26.5126133Z 2025-10-10T01:55:26.5126504Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5126832Z 2025-10-10T01:55:26.5127441Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:26.5127925Z 2025-10-10T01:55:26.5128201Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:26.5128478Z 2025-10-10T01:55:26.5128730Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5128995Z 2025-10-10T01:55:26.5129564Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:26.5130041Z 2025-10-10T01:55:26.5130340Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:26.5130635Z 2025-10-10T01:55:26.5130907Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5131177Z 2025-10-10T01:55:26.5131710Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:26.5132192Z 2025-10-10T01:55:26.5132495Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:26.5132820Z 2025-10-10T01:55:26.5133093Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5133378Z 2025-10-10T01:55:26.5133818Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:26.5134233Z 2025-10-10T01:55:26.5134578Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:26.5134916Z 2025-10-10T01:55:26.5135198Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5135474Z 2025-10-10T01:55:26.5135891Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:26.5136229Z 2025-10-10T01:55:26.5136587Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:26.5136921Z 2025-10-10T01:55:26.5137300Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:26.5137656Z 2025-10-10T01:55:26.5137993Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:26.5138374Z 2025-10-10T01:55:26.5138743Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:26.5139102Z 2025-10-10T01:55:26.5139475Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:26.5139885Z 2025-10-10T01:55:26.5140390Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:26.5140852Z 2025-10-10T01:55:26.5141196Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:26.5141528Z 2025-10-10T01:55:26.5142047Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:26.5142523Z 2025-10-10T01:55:26.5142898Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:26.5143259Z 2025-10-10T01:55:26.5143856Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:26.5144362Z 2025-10-10T01:55:26.5144659Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:26.5144950Z 2025-10-10T01:55:26.5145571Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:26.5146149Z 2025-10-10T01:55:26.5146512Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:26.5146867Z 2025-10-10T01:55:26.5147117Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5147387Z 2025-10-10T01:55:26.5147944Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:26.5148462Z 2025-10-10T01:55:26.5148755Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:26.5149049Z 2025-10-10T01:55:26.5149328Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5149602Z 2025-10-10T01:55:26.5150205Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:26.5150761Z 2025-10-10T01:55:26.5151095Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:26.5151380Z 2025-10-10T01:55:26.5151641Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5151917Z 2025-10-10T01:55:26.5152496Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:26.5153032Z 2025-10-10T01:55:26.5153363Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:26.5153709Z 2025-10-10T01:55:26.5153990Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5154260Z 2025-10-10T01:55:26.5154784Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:26.5155326Z 2025-10-10T01:55:26.5155610Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:26.5155898Z 2025-10-10T01:55:26.5156154Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5156423Z 2025-10-10T01:55:26.5156932Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:26.5157412Z 2025-10-10T01:55:26.5157716Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:26.5158024Z 2025-10-10T01:55:26.5158244Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:26.5158481Z 2025-10-10T01:55:26.5159124Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:26.5159580Z 2025-10-10T01:55:26.5159834Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] raise RuntimeError( 2025-10-10T01:55:26.5160087Z 2025-10-10T01:55:26.5160657Z (EngineCore_DP0 pid=9672) ERROR 10-10 01:55:26 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:26.5161390Z (EngineCore_DP0 pid=9672) Process EngineCore_DP0: 2025-10-10T01:55:26.5161781Z (EngineCore_DP0 pid=9672) Traceback (most recent call last): 2025-10-10T01:55:26.5162390Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:26.5162925Z (EngineCore_DP0 pid=9672) self.run() 2025-10-10T01:55:26.5163462Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:26.5164023Z (EngineCore_DP0 pid=9672) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:26.5164680Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:26.5165223Z (EngineCore_DP0 pid=9672) raise e 2025-10-10T01:55:26.5165810Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:26.5166437Z (EngineCore_DP0 pid=9672) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:26.5166873Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5167527Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:26.5168165Z (EngineCore_DP0 pid=9672) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:26.5168821Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:26.5169433Z (EngineCore_DP0 pid=9672) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:26.5169924Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5170565Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:26.5171148Z (EngineCore_DP0 pid=9672) self._init_executor() 2025-10-10T01:55:26.5171850Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:26.5172547Z (EngineCore_DP0 pid=9672) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:26.5173275Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:26.5173980Z (EngineCore_DP0 pid=9672) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:26.5174469Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5175109Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:26.5175690Z (EngineCore_DP0 pid=9672) return func(*args, **kwargs) 2025-10-10T01:55:26.5176077Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5176754Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:26.5177379Z (EngineCore_DP0 pid=9672) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:26.5177805Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5178467Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:26.5179120Z (EngineCore_DP0 pid=9672) module = importlib.import_module(module_name) 2025-10-10T01:55:26.5179562Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5180136Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:26.5180765Z (EngineCore_DP0 pid=9672) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:26.5181256Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5181750Z (EngineCore_DP0 pid=9672) File "", line 1387, in _gcd_import 2025-10-10T01:55:26.5182309Z (EngineCore_DP0 pid=9672) File "", line 1360, in _find_and_load 2025-10-10T01:55:26.5182894Z (EngineCore_DP0 pid=9672) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:26.5183484Z (EngineCore_DP0 pid=9672) File "", line 935, in _load_unlocked 2025-10-10T01:55:26.5184095Z (EngineCore_DP0 pid=9672) File "", line 999, in exec_module 2025-10-10T01:55:26.5184709Z (EngineCore_DP0 pid=9672) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:26.5185439Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:26.5186108Z (EngineCore_DP0 pid=9672) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:26.5186820Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:26.5187598Z (EngineCore_DP0 pid=9672) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:26.5188386Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:26.5189104Z (EngineCore_DP0 pid=9672) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:26.5189886Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:26.5190687Z (EngineCore_DP0 pid=9672) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:26.5191163Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5191916Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:26.5192734Z (EngineCore_DP0 pid=9672) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:26.5193171Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5193991Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:26.5194702Z (EngineCore_DP0 pid=9672) return _is_fa2_supported(device)[0] 2025-10-10T01:55:26.5195110Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5195815Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:26.5196717Z (EngineCore_DP0 pid=9672) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:26.5197171Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5197828Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:26.5198464Z (EngineCore_DP0 pid=9672) prop = get_device_properties(device) 2025-10-10T01:55:26.5198878Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:26.5199589Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:26.5200262Z (EngineCore_DP0 pid=9672) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:26.5200651Z (EngineCore_DP0 pid=9672) ^^^^^^^^^^^^ 2025-10-10T01:55:26.5201228Z (EngineCore_DP0 pid=9672) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:26.5201788Z (EngineCore_DP0 pid=9672) raise RuntimeError( 2025-10-10T01:55:26.5202547Z (EngineCore_DP0 pid=9672) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:26.9137309Z FAILED 2025-10-10T01:55:26.9266433Z models/test_initialization.py::test_can_initialize_large_subset[RobertaModel] Fork a new process to run a test 9676 2025-10-10T01:55:26.9278433Z Fork a new process to run a test 0 2025-10-10T01:55:26.9556799Z INFO 10-10 01:55:26 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='RobertaModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'sentence-transformers/stsb-roberta-base-v2'} 2025-10-10T01:55:27.1035516Z 2025-10-10T01:55:27.1036714Z config.json: 0% 0.00/675 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 72, 'local_cache_dir': None} 2025-10-10T01:55:35.0402214Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:35.0402633Z 2025-10-10T01:55:35.0403023Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:35.0403402Z 2025-10-10T01:55:35.0404338Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:35.0404950Z 2025-10-10T01:55:35.0405355Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:35.0405730Z 2025-10-10T01:55:35.0406090Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0406450Z 2025-10-10T01:55:35.0407067Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:35.0407634Z 2025-10-10T01:55:35.0408080Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:35.0408494Z 2025-10-10T01:55:35.0409109Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:35.0409673Z 2025-10-10T01:55:35.0410068Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:35.0410443Z 2025-10-10T01:55:35.0410791Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0411135Z 2025-10-10T01:55:35.0411644Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:35.0412111Z 2025-10-10T01:55:35.0412438Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] self._init_executor() 2025-10-10T01:55:35.0412712Z 2025-10-10T01:55:35.0413269Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:35.0413923Z 2025-10-10T01:55:35.0414489Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:35.0414975Z 2025-10-10T01:55:35.0415573Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:35.0416262Z 2025-10-10T01:55:35.0416928Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:35.0417273Z 2025-10-10T01:55:35.0417754Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0418137Z 2025-10-10T01:55:35.0418650Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:35.0419109Z 2025-10-10T01:55:35.0419376Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:35.0419648Z 2025-10-10T01:55:35.0419892Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0420153Z 2025-10-10T01:55:35.0420665Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:35.0421161Z 2025-10-10T01:55:35.0421449Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:35.0421791Z 2025-10-10T01:55:35.0422059Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0422325Z 2025-10-10T01:55:35.0422862Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:35.0423342Z 2025-10-10T01:55:35.0423645Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:35.0423943Z 2025-10-10T01:55:35.0424220Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0424489Z 2025-10-10T01:55:35.0424922Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:35.0425337Z 2025-10-10T01:55:35.0425680Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:35.0426013Z 2025-10-10T01:55:35.0426296Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0426584Z 2025-10-10T01:55:35.0426931Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:35.0427271Z 2025-10-10T01:55:35.0427614Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:35.0427945Z 2025-10-10T01:55:35.0428360Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:35.0428725Z 2025-10-10T01:55:35.0429076Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:35.0429404Z 2025-10-10T01:55:35.0429766Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:35.0430121Z 2025-10-10T01:55:35.0430492Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:35.0430893Z 2025-10-10T01:55:35.0431392Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:35.0431893Z 2025-10-10T01:55:35.0432260Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:35.0432622Z 2025-10-10T01:55:35.0433149Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:35.0433631Z 2025-10-10T01:55:35.0433999Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:35.0434354Z 2025-10-10T01:55:35.0434913Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:35.0435421Z 2025-10-10T01:55:35.0435718Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:35.0436017Z 2025-10-10T01:55:35.0436694Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:35.0437307Z 2025-10-10T01:55:35.0437686Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:35.0438035Z 2025-10-10T01:55:35.0438289Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0438557Z 2025-10-10T01:55:35.0439254Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:35.0439778Z 2025-10-10T01:55:35.0440077Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:35.0440378Z 2025-10-10T01:55:35.0440647Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0440917Z 2025-10-10T01:55:35.0441519Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:35.0442063Z 2025-10-10T01:55:35.0442354Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:35.0442647Z 2025-10-10T01:55:35.0442902Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0443179Z 2025-10-10T01:55:35.0443818Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:35.0444364Z 2025-10-10T01:55:35.0444683Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:35.0444993Z 2025-10-10T01:55:35.0445270Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0445545Z 2025-10-10T01:55:35.0446072Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:35.0446591Z 2025-10-10T01:55:35.0446877Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:35.0447161Z 2025-10-10T01:55:35.0447428Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0447753Z 2025-10-10T01:55:35.0448276Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:35.0448769Z 2025-10-10T01:55:35.0449075Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:35.0449391Z 2025-10-10T01:55:35.0449616Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:35.0449865Z 2025-10-10T01:55:35.0450342Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:35.0450803Z 2025-10-10T01:55:35.0451060Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] raise RuntimeError( 2025-10-10T01:55:35.0451318Z 2025-10-10T01:55:35.0451923Z (EngineCore_DP0 pid=9735) ERROR 10-10 01:55:35 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:35.0452622Z (EngineCore_DP0 pid=9735) Process EngineCore_DP0: 2025-10-10T01:55:35.0453032Z (EngineCore_DP0 pid=9735) Traceback (most recent call last): 2025-10-10T01:55:35.0453631Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:35.0454166Z (EngineCore_DP0 pid=9735) self.run() 2025-10-10T01:55:35.0454700Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:35.0455260Z (EngineCore_DP0 pid=9735) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:35.0455931Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:35.0456473Z (EngineCore_DP0 pid=9735) raise e 2025-10-10T01:55:35.0457058Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:35.0457690Z (EngineCore_DP0 pid=9735) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:35.0458125Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0458744Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:35.0459375Z (EngineCore_DP0 pid=9735) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:35.0460082Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:35.0460711Z (EngineCore_DP0 pid=9735) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:35.0461159Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0461813Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:35.0462393Z (EngineCore_DP0 pid=9735) self._init_executor() 2025-10-10T01:55:35.0463090Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:35.0463774Z (EngineCore_DP0 pid=9735) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:35.0464608Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:35.0465337Z (EngineCore_DP0 pid=9735) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:35.0465831Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0466461Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:35.0467053Z (EngineCore_DP0 pid=9735) return func(*args, **kwargs) 2025-10-10T01:55:35.0467431Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0468064Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:35.0468692Z (EngineCore_DP0 pid=9735) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:35.0469158Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0469843Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:35.0470510Z (EngineCore_DP0 pid=9735) module = importlib.import_module(module_name) 2025-10-10T01:55:35.0470951Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0471533Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:35.0472152Z (EngineCore_DP0 pid=9735) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:35.0472648Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0473148Z (EngineCore_DP0 pid=9735) File "", line 1387, in _gcd_import 2025-10-10T01:55:35.0473704Z (EngineCore_DP0 pid=9735) File "", line 1360, in _find_and_load 2025-10-10T01:55:35.0474289Z (EngineCore_DP0 pid=9735) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:35.0474869Z (EngineCore_DP0 pid=9735) File "", line 935, in _load_unlocked 2025-10-10T01:55:35.0475440Z (EngineCore_DP0 pid=9735) File "", line 999, in exec_module 2025-10-10T01:55:35.0476046Z (EngineCore_DP0 pid=9735) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:35.0476819Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:35.0477513Z (EngineCore_DP0 pid=9735) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:35.0478228Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:35.0478938Z (EngineCore_DP0 pid=9735) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:35.0479769Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:35.0480498Z (EngineCore_DP0 pid=9735) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:35.0481306Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:35.0482164Z (EngineCore_DP0 pid=9735) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:35.0482644Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0483351Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:35.0484028Z (EngineCore_DP0 pid=9735) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:35.0484463Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0485210Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:35.0485903Z (EngineCore_DP0 pid=9735) return _is_fa2_supported(device)[0] 2025-10-10T01:55:35.0486307Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0487070Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:35.0487790Z (EngineCore_DP0 pid=9735) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:35.0488238Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0488914Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:35.0489556Z (EngineCore_DP0 pid=9735) prop = get_device_properties(device) 2025-10-10T01:55:35.0489969Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:35.0490631Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:35.0491316Z (EngineCore_DP0 pid=9735) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:35.0491712Z (EngineCore_DP0 pid=9735) ^^^^^^^^^^^^ 2025-10-10T01:55:35.0492288Z (EngineCore_DP0 pid=9735) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:35.0492841Z (EngineCore_DP0 pid=9735) raise RuntimeError( 2025-10-10T01:55:35.0493526Z (EngineCore_DP0 pid=9735) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:35.4491430Z FAILED 2025-10-10T01:55:35.4623368Z models/test_initialization.py::test_can_initialize_large_subset[DeepseekV2ForCausalLM] Fork a new process to run a test 9739 2025-10-10T01:55:35.4634113Z Fork a new process to run a test 0 2025-10-10T01:55:35.4911744Z INFO 10-10 01:55:35 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DeepseekV2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'deepseek-ai/DeepSeek-V2-Lite-Chat'} 2025-10-10T01:55:35.5835770Z 2025-10-10T01:55:35.5837946Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:35.5838375Z config.json: 1.52kB [00:00, 7.73MB/s] 2025-10-10T01:55:35.6766919Z 2025-10-10T01:55:35.6768804Z configuration_deepseek.py: 0.00B [00:00, ?B/s] 2025-10-10T01:55:35.6769319Z configuration_deepseek.py: 10.3kB [00:00, 66.9MB/s] 2025-10-10T01:55:35.6859748Z A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat: 2025-10-10T01:55:35.6860320Z - configuration_deepseek.py 2025-10-10T01:55:35.6860947Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:55:35.7188933Z INFO 10-10 01:55:35 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:55:35.8858727Z INFO 10-10 01:55:35 [model.py:551] Resolved architecture: DeepseekV2ForCausalLM 2025-10-10T01:55:35.8859365Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:55:35.9112260Z INFO 10-10 01:55:35 [model.py:1545] Using max model len 163840 2025-10-10T01:55:35.9113922Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:55:35.9749389Z INFO 10-10 01:55:35 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:55:36.1554412Z 2025-10-10T01:55:36.1556066Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:36.1556429Z tokenizer_config.json: 1.28kB [00:00, 10.9MB/s] 2025-10-10T01:55:36.3155437Z 2025-10-10T01:55:36.3351083Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:36.3351518Z tokenizer.json: 4.61MB [00:00, 236MB/s] 2025-10-10T01:55:36.7388887Z 2025-10-10T01:55:36.7389745Z generation_config.json: 0% 0.00/181 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:55:36.9024258Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:36.9024837Z 2025-10-10T01:55:36.9025427Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:36.9025810Z 2025-10-10T01:55:36.9026522Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:36.9027333Z 2025-10-10T01:55:36.9027751Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:36.9028129Z 2025-10-10T01:55:36.9028483Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9028822Z 2025-10-10T01:55:36.9029438Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:36.9029993Z 2025-10-10T01:55:36.9030685Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:36.9031143Z 2025-10-10T01:55:36.9031889Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:36.9032461Z 2025-10-10T01:55:36.9032866Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:36.9033196Z 2025-10-10T01:55:36.9033472Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9033994Z 2025-10-10T01:55:36.9034893Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:36.9035765Z 2025-10-10T01:55:36.9036218Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] self._init_executor() 2025-10-10T01:55:36.9036609Z 2025-10-10T01:55:36.9037449Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:36.9038077Z 2025-10-10T01:55:36.9038424Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:36.9038750Z 2025-10-10T01:55:36.9039409Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:36.9039923Z 2025-10-10T01:55:36.9040271Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:36.9040600Z 2025-10-10T01:55:36.9040897Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9041290Z 2025-10-10T01:55:36.9041796Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:36.9042253Z 2025-10-10T01:55:36.9042530Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:36.9042806Z 2025-10-10T01:55:36.9043056Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9043317Z 2025-10-10T01:55:36.9043917Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:36.9044394Z 2025-10-10T01:55:36.9044679Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:36.9045012Z 2025-10-10T01:55:36.9045281Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9045550Z 2025-10-10T01:55:36.9046080Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:36.9046556Z 2025-10-10T01:55:36.9046864Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:36.9047171Z 2025-10-10T01:55:36.9047450Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9047721Z 2025-10-10T01:55:36.9048169Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:36.9048579Z 2025-10-10T01:55:36.9048957Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:36.9049288Z 2025-10-10T01:55:36.9049574Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9049853Z 2025-10-10T01:55:36.9050192Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:36.9050521Z 2025-10-10T01:55:36.9050867Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:36.9051197Z 2025-10-10T01:55:36.9051574Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:36.9051929Z 2025-10-10T01:55:36.9052278Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:36.9052608Z 2025-10-10T01:55:36.9052970Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:36.9053319Z 2025-10-10T01:55:36.9053695Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:36.9054054Z 2025-10-10T01:55:36.9054565Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:36.9055044Z 2025-10-10T01:55:36.9055450Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:36.9055802Z 2025-10-10T01:55:36.9056325Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:36.9056805Z 2025-10-10T01:55:36.9057173Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:36.9057535Z 2025-10-10T01:55:36.9058084Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:36.9058625Z 2025-10-10T01:55:36.9058920Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:36.9059212Z 2025-10-10T01:55:36.9059867Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:36.9060493Z 2025-10-10T01:55:36.9060867Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:36.9061216Z 2025-10-10T01:55:36.9061470Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9061735Z 2025-10-10T01:55:36.9062304Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:36.9062841Z 2025-10-10T01:55:36.9063137Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:36.9063437Z 2025-10-10T01:55:36.9063708Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9064022Z 2025-10-10T01:55:36.9064618Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:36.9065163Z 2025-10-10T01:55:36.9065449Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:36.9065730Z 2025-10-10T01:55:36.9065995Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9066259Z 2025-10-10T01:55:36.9066846Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:36.9067382Z 2025-10-10T01:55:36.9067703Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:36.9068014Z 2025-10-10T01:55:36.9068284Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9068562Z 2025-10-10T01:55:36.9069079Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:36.9069566Z 2025-10-10T01:55:36.9069842Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:36.9070134Z 2025-10-10T01:55:36.9070390Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9070655Z 2025-10-10T01:55:36.9071229Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:36.9071716Z 2025-10-10T01:55:36.9072035Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:36.9072337Z 2025-10-10T01:55:36.9072575Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:36.9072812Z 2025-10-10T01:55:36.9073295Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:36.9073788Z 2025-10-10T01:55:36.9074036Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] raise RuntimeError( 2025-10-10T01:55:36.9074295Z 2025-10-10T01:55:36.9074901Z (EngineCore_DP0 pid=9747) ERROR 10-10 01:55:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:36.9075591Z (EngineCore_DP0 pid=9747) Process EngineCore_DP0: 2025-10-10T01:55:36.9075986Z (EngineCore_DP0 pid=9747) Traceback (most recent call last): 2025-10-10T01:55:36.9076588Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:36.9077104Z (EngineCore_DP0 pid=9747) self.run() 2025-10-10T01:55:36.9077629Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:36.9078188Z (EngineCore_DP0 pid=9747) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:36.9078849Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:36.9079529Z (EngineCore_DP0 pid=9747) raise e 2025-10-10T01:55:36.9080117Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:36.9080749Z (EngineCore_DP0 pid=9747) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:36.9081186Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9081806Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:36.9082445Z (EngineCore_DP0 pid=9747) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:36.9083132Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:36.9083766Z (EngineCore_DP0 pid=9747) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:36.9084221Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9084871Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:36.9085457Z (EngineCore_DP0 pid=9747) self._init_executor() 2025-10-10T01:55:36.9086121Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:36.9086801Z (EngineCore_DP0 pid=9747) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:36.9087588Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:36.9088329Z (EngineCore_DP0 pid=9747) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:36.9088828Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9089475Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:36.9090059Z (EngineCore_DP0 pid=9747) return func(*args, **kwargs) 2025-10-10T01:55:36.9090440Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9091121Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:36.9091749Z (EngineCore_DP0 pid=9747) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:36.9092220Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9092885Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:36.9093556Z (EngineCore_DP0 pid=9747) module = importlib.import_module(module_name) 2025-10-10T01:55:36.9094017Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9094596Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:36.9095233Z (EngineCore_DP0 pid=9747) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:36.9095730Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9096412Z (EngineCore_DP0 pid=9747) File "", line 1387, in _gcd_import 2025-10-10T01:55:36.9097084Z (EngineCore_DP0 pid=9747) File "", line 1360, in _find_and_load 2025-10-10T01:55:36.9097701Z (EngineCore_DP0 pid=9747) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:36.9098282Z (EngineCore_DP0 pid=9747) File "", line 935, in _load_unlocked 2025-10-10T01:55:36.9098854Z (EngineCore_DP0 pid=9747) File "", line 999, in exec_module 2025-10-10T01:55:36.9099447Z (EngineCore_DP0 pid=9747) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:36.9100213Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:36.9100897Z (EngineCore_DP0 pid=9747) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:36.9101623Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:36.9102352Z (EngineCore_DP0 pid=9747) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:36.9103128Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:36.9103788Z (EngineCore_DP0 pid=9747) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:36.9104580Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:36.9105385Z (EngineCore_DP0 pid=9747) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:36.9105950Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9106653Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:36.9107324Z (EngineCore_DP0 pid=9747) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:36.9107762Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9108502Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:36.9109263Z (EngineCore_DP0 pid=9747) return _is_fa2_supported(device)[0] 2025-10-10T01:55:36.9109670Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9110378Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:36.9111165Z (EngineCore_DP0 pid=9747) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:36.9111619Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9112293Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:36.9112931Z (EngineCore_DP0 pid=9747) prop = get_device_properties(device) 2025-10-10T01:55:36.9113345Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:36.9114012Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:36.9114677Z (EngineCore_DP0 pid=9747) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:36.9115115Z (EngineCore_DP0 pid=9747) ^^^^^^^^^^^^ 2025-10-10T01:55:36.9115699Z (EngineCore_DP0 pid=9747) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:36.9116258Z (EngineCore_DP0 pid=9747) raise RuntimeError( 2025-10-10T01:55:36.9116933Z (EngineCore_DP0 pid=9747) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:37.3084275Z FAILED 2025-10-10T01:55:37.3214226Z models/test_initialization.py::test_can_initialize_large_subset[SeedOssForCausalLM] Fork a new process to run a test 9751 2025-10-10T01:55:37.3224565Z Fork a new process to run a test 0 2025-10-10T01:55:37.3225782Z Model is not available online 2025-10-10T01:55:37.6262479Z PASSED 2025-10-10T01:55:37.6390104Z models/test_initialization.py::test_can_initialize_large_subset[OrionForCausalLM] Fork a new process to run a test 9752 2025-10-10T01:55:37.6401337Z Fork a new process to run a test 0 2025-10-10T01:55:37.6679766Z INFO 10-10 01:55:37 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='OrionForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'OrionStarAI/Orion-14B-Chat'} 2025-10-10T01:55:37.8287980Z 2025-10-10T01:55:37.8289317Z config.json: 0% 0.00/771 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:55:45.9122812Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:45.9123408Z 2025-10-10T01:55:45.9123933Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:45.9124787Z 2025-10-10T01:55:45.9125782Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:45.9126647Z 2025-10-10T01:55:45.9127223Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:45.9127797Z 2025-10-10T01:55:45.9128340Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9128892Z 2025-10-10T01:55:45.9129870Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:45.9130782Z 2025-10-10T01:55:45.9131473Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:45.9132133Z 2025-10-10T01:55:45.9133272Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:45.9134183Z 2025-10-10T01:55:45.9134840Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:45.9135435Z 2025-10-10T01:55:45.9135982Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9136506Z 2025-10-10T01:55:45.9137476Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:45.9138355Z 2025-10-10T01:55:45.9138832Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] self._init_executor() 2025-10-10T01:55:45.9139336Z 2025-10-10T01:55:45.9140313Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:45.9141128Z 2025-10-10T01:55:45.9141658Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:45.9142233Z 2025-10-10T01:55:45.9143195Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:45.9144040Z 2025-10-10T01:55:45.9144456Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:45.9144800Z 2025-10-10T01:55:45.9145245Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9145538Z 2025-10-10T01:55:45.9146067Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:45.9146526Z 2025-10-10T01:55:45.9146799Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:45.9147065Z 2025-10-10T01:55:45.9147320Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9147672Z 2025-10-10T01:55:45.9148188Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:45.9148675Z 2025-10-10T01:55:45.9148963Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:45.9149299Z 2025-10-10T01:55:45.9149565Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9149836Z 2025-10-10T01:55:45.9150358Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:45.9150855Z 2025-10-10T01:55:45.9151162Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:45.9151460Z 2025-10-10T01:55:45.9151734Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9152006Z 2025-10-10T01:55:45.9152450Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:45.9152877Z 2025-10-10T01:55:45.9153273Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:45.9153607Z 2025-10-10T01:55:45.9153908Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9154194Z 2025-10-10T01:55:45.9154540Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:45.9154878Z 2025-10-10T01:55:45.9155223Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:45.9155560Z 2025-10-10T01:55:45.9155931Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:45.9156291Z 2025-10-10T01:55:45.9156636Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:45.9156964Z 2025-10-10T01:55:45.9157339Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:45.9157696Z 2025-10-10T01:55:45.9158079Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:45.9158437Z 2025-10-10T01:55:45.9158939Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:45.9159552Z 2025-10-10T01:55:45.9159952Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:45.9160296Z 2025-10-10T01:55:45.9160814Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:45.9161294Z 2025-10-10T01:55:45.9161663Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:45.9162023Z 2025-10-10T01:55:45.9162569Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:45.9163118Z 2025-10-10T01:55:45.9163411Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:45.9163706Z 2025-10-10T01:55:45.9164376Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:45.9164972Z 2025-10-10T01:55:45.9165342Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:45.9165694Z 2025-10-10T01:55:45.9165952Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9166214Z 2025-10-10T01:55:45.9166770Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:45.9167288Z 2025-10-10T01:55:45.9167582Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:45.9167880Z 2025-10-10T01:55:45.9168188Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9168467Z 2025-10-10T01:55:45.9169062Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:45.9169612Z 2025-10-10T01:55:45.9169892Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:45.9170176Z 2025-10-10T01:55:45.9170438Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9170701Z 2025-10-10T01:55:45.9171289Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:45.9171814Z 2025-10-10T01:55:45.9172139Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:45.9172447Z 2025-10-10T01:55:45.9172716Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9172992Z 2025-10-10T01:55:45.9173506Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:45.9174025Z 2025-10-10T01:55:45.9174307Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:45.9174597Z 2025-10-10T01:55:45.9174853Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9175200Z 2025-10-10T01:55:45.9175724Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:45.9176207Z 2025-10-10T01:55:45.9176521Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:45.9176819Z 2025-10-10T01:55:45.9177045Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:45.9177330Z 2025-10-10T01:55:45.9177827Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:45.9178271Z 2025-10-10T01:55:45.9178513Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] raise RuntimeError( 2025-10-10T01:55:45.9178818Z 2025-10-10T01:55:45.9179388Z (EngineCore_DP0 pid=9830) ERROR 10-10 01:55:45 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:45.9180080Z (EngineCore_DP0 pid=9830) Process EngineCore_DP0: 2025-10-10T01:55:45.9180519Z (EngineCore_DP0 pid=9830) Traceback (most recent call last): 2025-10-10T01:55:45.9181130Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:45.9181647Z (EngineCore_DP0 pid=9830) self.run() 2025-10-10T01:55:45.9182169Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:45.9182728Z (EngineCore_DP0 pid=9830) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:45.9183388Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:45.9183976Z (EngineCore_DP0 pid=9830) raise e 2025-10-10T01:55:45.9184568Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:45.9185195Z (EngineCore_DP0 pid=9830) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:45.9185628Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9186251Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:45.9186885Z (EngineCore_DP0 pid=9830) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:45.9187540Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:45.9188161Z (EngineCore_DP0 pid=9830) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:45.9188606Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9189266Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:45.9189853Z (EngineCore_DP0 pid=9830) self._init_executor() 2025-10-10T01:55:45.9190517Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:45.9191200Z (EngineCore_DP0 pid=9830) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:45.9191962Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:45.9192673Z (EngineCore_DP0 pid=9830) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:45.9193170Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9193813Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:45.9194399Z (EngineCore_DP0 pid=9830) return func(*args, **kwargs) 2025-10-10T01:55:45.9194831Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9195477Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:45.9196366Z (EngineCore_DP0 pid=9830) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:45.9196961Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9197663Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:45.9198327Z (EngineCore_DP0 pid=9830) module = importlib.import_module(module_name) 2025-10-10T01:55:45.9198767Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9199422Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:45.9200054Z (EngineCore_DP0 pid=9830) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:45.9200547Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9201048Z (EngineCore_DP0 pid=9830) File "", line 1387, in _gcd_import 2025-10-10T01:55:45.9201667Z (EngineCore_DP0 pid=9830) File "", line 1360, in _find_and_load 2025-10-10T01:55:45.9202264Z (EngineCore_DP0 pid=9830) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:45.9202845Z (EngineCore_DP0 pid=9830) File "", line 935, in _load_unlocked 2025-10-10T01:55:45.9203414Z (EngineCore_DP0 pid=9830) File "", line 999, in exec_module 2025-10-10T01:55:45.9204020Z (EngineCore_DP0 pid=9830) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:45.9204749Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:45.9205442Z (EngineCore_DP0 pid=9830) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:45.9206178Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:45.9206935Z (EngineCore_DP0 pid=9830) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:45.9207727Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:45.9208405Z (EngineCore_DP0 pid=9830) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:45.9209188Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:45.9210068Z (EngineCore_DP0 pid=9830) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:45.9210557Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9211244Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:45.9211924Z (EngineCore_DP0 pid=9830) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:45.9212353Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9213091Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:45.9213850Z (EngineCore_DP0 pid=9830) return _is_fa2_supported(device)[0] 2025-10-10T01:55:45.9214262Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9215022Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:45.9215734Z (EngineCore_DP0 pid=9830) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:45.9216181Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9216851Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:45.9217484Z (EngineCore_DP0 pid=9830) prop = get_device_properties(device) 2025-10-10T01:55:45.9217897Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:45.9218548Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:45.9219221Z (EngineCore_DP0 pid=9830) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:45.9219655Z (EngineCore_DP0 pid=9830) ^^^^^^^^^^^^ 2025-10-10T01:55:45.9227318Z (EngineCore_DP0 pid=9830) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:45.9228154Z (EngineCore_DP0 pid=9830) raise RuntimeError( 2025-10-10T01:55:45.9228890Z (EngineCore_DP0 pid=9830) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:46.3232417Z FAILED 2025-10-10T01:55:46.3363463Z models/test_initialization.py::test_can_initialize_large_subset[ModernBertModel] Fork a new process to run a test 9834 2025-10-10T01:55:46.3374180Z Fork a new process to run a test 0 2025-10-10T01:55:46.3643630Z INFO 10-10 01:55:46 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ModernBertModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Alibaba-NLP/gte-modernbert-base'} 2025-10-10T01:55:46.5057992Z 2025-10-10T01:55:46.5060392Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:46.5060722Z config.json: 1.18kB [00:00, 5.93MB/s] 2025-10-10T01:55:53.2710166Z 2025-10-10T01:55:53.2710702Z modules.json: 0% 0.00/229 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:55:54.4379691Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] EngineCore failed to start. 2025-10-10T01:55:54.4380159Z 2025-10-10T01:55:54.4380721Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] Traceback (most recent call last): 2025-10-10T01:55:54.4381092Z 2025-10-10T01:55:54.4381792Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:54.4382722Z 2025-10-10T01:55:54.4383138Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:54.4383521Z 2025-10-10T01:55:54.4383884Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4384363Z 2025-10-10T01:55:54.4385040Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:54.4385788Z 2025-10-10T01:55:54.4386362Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:54.4386860Z 2025-10-10T01:55:54.4387560Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:54.4388140Z 2025-10-10T01:55:54.4388537Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:54.4389053Z 2025-10-10T01:55:54.4389405Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4389912Z 2025-10-10T01:55:54.4390969Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:54.4391863Z 2025-10-10T01:55:54.4392228Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] self._init_executor() 2025-10-10T01:55:54.4392608Z 2025-10-10T01:55:54.4393185Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:54.4393698Z 2025-10-10T01:55:54.4394029Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:54.4394349Z 2025-10-10T01:55:54.4394904Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:54.4395436Z 2025-10-10T01:55:54.4395782Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:54.4396425Z 2025-10-10T01:55:54.4396742Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4397034Z 2025-10-10T01:55:54.4397523Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:54.4397993Z 2025-10-10T01:55:54.4398260Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:55:54.4398527Z 2025-10-10T01:55:54.4398921Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4399346Z 2025-10-10T01:55:54.4399885Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:54.4400358Z 2025-10-10T01:55:54.4400655Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:54.4400941Z 2025-10-10T01:55:54.4401204Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4401574Z 2025-10-10T01:55:54.4402113Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:54.4402681Z 2025-10-10T01:55:54.4403004Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:55:54.4403311Z 2025-10-10T01:55:54.4403592Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4403863Z 2025-10-10T01:55:54.4404306Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:54.4404715Z 2025-10-10T01:55:54.4405057Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:54.4405389Z 2025-10-10T01:55:54.4405679Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4405967Z 2025-10-10T01:55:54.4406325Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:55:54.4406783Z 2025-10-10T01:55:54.4407147Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:55:54.4407489Z 2025-10-10T01:55:54.4407863Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:54.4408227Z 2025-10-10T01:55:54.4408568Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:55:54.4408914Z 2025-10-10T01:55:54.4409283Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:55:54.4409632Z 2025-10-10T01:55:54.4410023Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:54.4410389Z 2025-10-10T01:55:54.4410902Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:54.4411378Z 2025-10-10T01:55:54.4411740Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:54.4412072Z 2025-10-10T01:55:54.4412595Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:54.4413080Z 2025-10-10T01:55:54.4413449Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:54.4413861Z 2025-10-10T01:55:54.4414415Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:54.4414926Z 2025-10-10T01:55:54.4415217Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:55:54.4415521Z 2025-10-10T01:55:54.4416142Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:54.4416756Z 2025-10-10T01:55:54.4417122Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:54.4417468Z 2025-10-10T01:55:54.4417728Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4418042Z 2025-10-10T01:55:54.4418619Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:54.4419143Z 2025-10-10T01:55:54.4419443Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:55:54.4419733Z 2025-10-10T01:55:54.4420004Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4420279Z 2025-10-10T01:55:54.4420889Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:54.4421447Z 2025-10-10T01:55:54.4421733Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:55:54.4422070Z 2025-10-10T01:55:54.4422337Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4422615Z 2025-10-10T01:55:54.4423195Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:54.4423722Z 2025-10-10T01:55:54.4424045Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:54.4424357Z 2025-10-10T01:55:54.4424636Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4424907Z 2025-10-10T01:55:54.4425438Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:54.4425933Z 2025-10-10T01:55:54.4426219Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:55:54.4426502Z 2025-10-10T01:55:54.4426763Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4427035Z 2025-10-10T01:55:54.4427550Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:54.4428036Z 2025-10-10T01:55:54.4428341Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:55:54.4428653Z 2025-10-10T01:55:54.4428943Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:55:54.4429190Z 2025-10-10T01:55:54.4429670Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:54.4430133Z 2025-10-10T01:55:54.4430376Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] raise RuntimeError( 2025-10-10T01:55:54.4430638Z 2025-10-10T01:55:54.4431201Z (EngineCore_DP0 pid=9893) ERROR 10-10 01:55:54 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:54.4431978Z (EngineCore_DP0 pid=9893) Process EngineCore_DP0: 2025-10-10T01:55:54.4432373Z (EngineCore_DP0 pid=9893) Traceback (most recent call last): 2025-10-10T01:55:54.4432978Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:55:54.4433578Z (EngineCore_DP0 pid=9893) self.run() 2025-10-10T01:55:54.4434102Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:55:54.4434659Z (EngineCore_DP0 pid=9893) self._target(*self._args, **self._kwargs) 2025-10-10T01:55:54.4435337Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:55:54.4435883Z (EngineCore_DP0 pid=9893) raise e 2025-10-10T01:55:54.4436464Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:55:54.4437092Z (EngineCore_DP0 pid=9893) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:55:54.4437543Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4438204Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:55:54.4438840Z (EngineCore_DP0 pid=9893) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:55:54.4439590Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:55:54.4440207Z (EngineCore_DP0 pid=9893) self.model_executor = executor_class(vllm_config) 2025-10-10T01:55:54.4440651Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4441298Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:55:54.4441928Z (EngineCore_DP0 pid=9893) self._init_executor() 2025-10-10T01:55:54.4442603Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:55:54.4443315Z (EngineCore_DP0 pid=9893) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:55:54.4444034Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:55:54.4444747Z (EngineCore_DP0 pid=9893) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:55:54.4445257Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4445909Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:55:54.4446567Z (EngineCore_DP0 pid=9893) return func(*args, **kwargs) 2025-10-10T01:55:54.4446961Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4447598Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:55:54.4448229Z (EngineCore_DP0 pid=9893) worker_class = resolve_obj_by_qualname( 2025-10-10T01:55:54.4448661Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4449372Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:55:54.4450028Z (EngineCore_DP0 pid=9893) module = importlib.import_module(module_name) 2025-10-10T01:55:54.4450467Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4451112Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:55:54.4451738Z (EngineCore_DP0 pid=9893) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:55:54.4452247Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4452747Z (EngineCore_DP0 pid=9893) File "", line 1387, in _gcd_import 2025-10-10T01:55:54.4453294Z (EngineCore_DP0 pid=9893) File "", line 1360, in _find_and_load 2025-10-10T01:55:54.4453881Z (EngineCore_DP0 pid=9893) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:55:54.4454458Z (EngineCore_DP0 pid=9893) File "", line 935, in _load_unlocked 2025-10-10T01:55:54.4455046Z (EngineCore_DP0 pid=9893) File "", line 999, in exec_module 2025-10-10T01:55:54.4455695Z (EngineCore_DP0 pid=9893) File "", line 488, in _call_with_frames_removed 2025-10-10T01:55:54.4456427Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:55:54.4457118Z (EngineCore_DP0 pid=9893) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:55:54.4457847Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:55:54.4458575Z (EngineCore_DP0 pid=9893) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:55:54.4459372Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:55:54.4460045Z (EngineCore_DP0 pid=9893) class FlashAttentionMetadataBuilder( 2025-10-10T01:55:54.4460835Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:55:54.4461678Z (EngineCore_DP0 pid=9893) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:55:54.4462165Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4462876Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:55:54.4463563Z (EngineCore_DP0 pid=9893) if not is_fa_version_supported(fa_version): 2025-10-10T01:55:54.4464049Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4464821Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:55:54.4465528Z (EngineCore_DP0 pid=9893) return _is_fa2_supported(device)[0] 2025-10-10T01:55:54.4465942Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4466655Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:55:54.4467408Z (EngineCore_DP0 pid=9893) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:55:54.4467863Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4468538Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:55:54.4469238Z (EngineCore_DP0 pid=9893) prop = get_device_properties(device) 2025-10-10T01:55:54.4469657Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:55:54.4470314Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:55:54.4470980Z (EngineCore_DP0 pid=9893) _lazy_init() # will define _get_device_properties 2025-10-10T01:55:54.4471393Z (EngineCore_DP0 pid=9893) ^^^^^^^^^^^^ 2025-10-10T01:55:54.4471978Z (EngineCore_DP0 pid=9893) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:55:54.4472541Z (EngineCore_DP0 pid=9893) raise RuntimeError( 2025-10-10T01:55:54.4473274Z (EngineCore_DP0 pid=9893) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:55:54.8449408Z FAILED 2025-10-10T01:55:54.8578224Z models/test_initialization.py::test_can_initialize_large_subset[MiMoForCausalLM] Fork a new process to run a test 9897 2025-10-10T01:55:54.8589745Z Fork a new process to run a test 0 2025-10-10T01:55:54.8869947Z INFO 10-10 01:55:54 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiMoForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'XiaomiMiMo/MiMo-7B-RL'} 2025-10-10T01:55:55.0051459Z 2025-10-10T01:55:55.0053641Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:55:55.0053954Z config.json: 1.04kB [00:00, 5.46MB/s] 2025-10-10T01:55:55.0752129Z 2025-10-10T01:55:55.0752923Z configuration_mimo.py: 0% 0.00/376 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:03.0825855Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:03.0826273Z 2025-10-10T01:56:03.0826647Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:03.0826959Z 2025-10-10T01:56:03.0827498Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:03.0828022Z 2025-10-10T01:56:03.0828346Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:03.0828652Z 2025-10-10T01:56:03.0829193Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0829507Z 2025-10-10T01:56:03.0830023Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:03.0830485Z 2025-10-10T01:56:03.0830811Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:03.0831135Z 2025-10-10T01:56:03.0831604Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:03.0832137Z 2025-10-10T01:56:03.0832453Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:03.0832762Z 2025-10-10T01:56:03.0833038Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0833404Z 2025-10-10T01:56:03.0833916Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:03.0834383Z 2025-10-10T01:56:03.0834650Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] self._init_executor() 2025-10-10T01:56:03.0834914Z 2025-10-10T01:56:03.0835464Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:03.0835960Z 2025-10-10T01:56:03.0836283Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:03.0836614Z 2025-10-10T01:56:03.0837726Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:03.0838415Z 2025-10-10T01:56:03.0839103Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:03.0839720Z 2025-10-10T01:56:03.0840045Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0840335Z 2025-10-10T01:56:03.0840834Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:03.0841296Z 2025-10-10T01:56:03.0841562Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:03.0841835Z 2025-10-10T01:56:03.0842095Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0842364Z 2025-10-10T01:56:03.0842921Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:03.0843403Z 2025-10-10T01:56:03.0843702Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:03.0843987Z 2025-10-10T01:56:03.0844253Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0844536Z 2025-10-10T01:56:03.0845059Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:03.0845548Z 2025-10-10T01:56:03.0845910Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:03.0846227Z 2025-10-10T01:56:03.0846544Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0846813Z 2025-10-10T01:56:03.0847257Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:03.0847663Z 2025-10-10T01:56:03.0847998Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:03.0848382Z 2025-10-10T01:56:03.0848670Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0848949Z 2025-10-10T01:56:03.0849289Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:03.0849661Z 2025-10-10T01:56:03.0850004Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:03.0850340Z 2025-10-10T01:56:03.0850717Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:03.0851069Z 2025-10-10T01:56:03.0851415Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:03.0851745Z 2025-10-10T01:56:03.0852105Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:03.0852449Z 2025-10-10T01:56:03.0852823Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:03.0853221Z 2025-10-10T01:56:03.0853728Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:03.0854194Z 2025-10-10T01:56:03.0854543Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:03.0854884Z 2025-10-10T01:56:03.0855399Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:03.0855880Z 2025-10-10T01:56:03.0856248Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:03.0856611Z 2025-10-10T01:56:03.0857157Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:03.0857655Z 2025-10-10T01:56:03.0857949Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:03.0858239Z 2025-10-10T01:56:03.0858881Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:03.0859450Z 2025-10-10T01:56:03.0859816Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:03.0860160Z 2025-10-10T01:56:03.0860453Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0860721Z 2025-10-10T01:56:03.0861279Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:03.0861804Z 2025-10-10T01:56:03.0862098Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:03.0862396Z 2025-10-10T01:56:03.0862662Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0862974Z 2025-10-10T01:56:03.0866654Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:03.0867291Z 2025-10-10T01:56:03.0867732Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:03.0868029Z 2025-10-10T01:56:03.0868297Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0868565Z 2025-10-10T01:56:03.0869149Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:03.0869686Z 2025-10-10T01:56:03.0870005Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:03.0870320Z 2025-10-10T01:56:03.0870593Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0870872Z 2025-10-10T01:56:03.0871465Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:03.0871955Z 2025-10-10T01:56:03.0872245Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:03.0872532Z 2025-10-10T01:56:03.0872799Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0873067Z 2025-10-10T01:56:03.0873583Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:03.0874108Z 2025-10-10T01:56:03.0874428Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:03.0874744Z 2025-10-10T01:56:03.0874972Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:03.0875221Z 2025-10-10T01:56:03.0875707Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:03.0876218Z 2025-10-10T01:56:03.0876509Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] raise RuntimeError( 2025-10-10T01:56:03.0876816Z 2025-10-10T01:56:03.0877499Z (EngineCore_DP0 pid=9955) ERROR 10-10 01:56:03 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:03.0878320Z (EngineCore_DP0 pid=9955) Process EngineCore_DP0: 2025-10-10T01:56:03.0878740Z (EngineCore_DP0 pid=9955) Traceback (most recent call last): 2025-10-10T01:56:03.0879495Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:03.0880022Z (EngineCore_DP0 pid=9955) self.run() 2025-10-10T01:56:03.0880564Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:03.0881133Z (EngineCore_DP0 pid=9955) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:03.0881807Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:03.0882406Z (EngineCore_DP0 pid=9955) raise e 2025-10-10T01:56:03.0883002Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:03.0883727Z (EngineCore_DP0 pid=9955) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:03.0884225Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0884853Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:03.0885490Z (EngineCore_DP0 pid=9955) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:03.0886211Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:03.0886947Z (EngineCore_DP0 pid=9955) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:03.0887480Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0888209Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:03.0888786Z (EngineCore_DP0 pid=9955) self._init_executor() 2025-10-10T01:56:03.0889490Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:03.0890182Z (EngineCore_DP0 pid=9955) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:03.0890900Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:03.0891608Z (EngineCore_DP0 pid=9955) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:03.0892103Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0892743Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:03.0893333Z (EngineCore_DP0 pid=9955) return func(*args, **kwargs) 2025-10-10T01:56:03.0893718Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0894375Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:03.0895023Z (EngineCore_DP0 pid=9955) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:03.0895444Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0896333Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:03.0897010Z (EngineCore_DP0 pid=9955) module = importlib.import_module(module_name) 2025-10-10T01:56:03.0897458Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0898057Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:03.0898676Z (EngineCore_DP0 pid=9955) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:03.0899178Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0899682Z (EngineCore_DP0 pid=9955) File "", line 1387, in _gcd_import 2025-10-10T01:56:03.0900234Z (EngineCore_DP0 pid=9955) File "", line 1360, in _find_and_load 2025-10-10T01:56:03.0900937Z (EngineCore_DP0 pid=9955) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:03.0901601Z (EngineCore_DP0 pid=9955) File "", line 935, in _load_unlocked 2025-10-10T01:56:03.0902235Z (EngineCore_DP0 pid=9955) File "", line 999, in exec_module 2025-10-10T01:56:03.0902838Z (EngineCore_DP0 pid=9955) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:03.0903580Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:03.0904265Z (EngineCore_DP0 pid=9955) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:03.0904985Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:03.0905713Z (EngineCore_DP0 pid=9955) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:03.0906495Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:03.0907224Z (EngineCore_DP0 pid=9955) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:03.0908016Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:03.0908830Z (EngineCore_DP0 pid=9955) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:03.0909304Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0909997Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:03.0910697Z (EngineCore_DP0 pid=9955) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:03.0911144Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0911893Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:03.0912582Z (EngineCore_DP0 pid=9955) return _is_fa2_supported(device)[0] 2025-10-10T01:56:03.0912992Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0913704Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:03.0914420Z (EngineCore_DP0 pid=9955) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:03.0914878Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0915566Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:03.0916207Z (EngineCore_DP0 pid=9955) prop = get_device_properties(device) 2025-10-10T01:56:03.0916636Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:03.0917313Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:03.0917981Z (EngineCore_DP0 pid=9955) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:03.0918430Z (EngineCore_DP0 pid=9955) ^^^^^^^^^^^^ 2025-10-10T01:56:03.0919064Z (EngineCore_DP0 pid=9955) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:03.0919678Z (EngineCore_DP0 pid=9955) raise RuntimeError( 2025-10-10T01:56:03.0920374Z (EngineCore_DP0 pid=9955) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:03.5267969Z FAILED 2025-10-10T01:56:03.5393254Z models/test_initialization.py::test_can_initialize_large_subset[TeleChat2ForCausalLM] Fork a new process to run a test 9959 2025-10-10T01:56:03.5405501Z Fork a new process to run a test 0 2025-10-10T01:56:03.5685402Z INFO 10-10 01:56:03 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='TeleChat2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Tele-AI/TeleChat2-3B'} 2025-10-10T01:56:03.7009749Z 2025-10-10T01:56:03.7011824Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:03.7012181Z config.json: 1.04kB [00:00, 5.17MB/s] 2025-10-10T01:56:03.8710519Z 2025-10-10T01:56:03.8712181Z configuration_telechat2.py: 0.00B [00:00, ?B/s] 2025-10-10T01:56:03.8712780Z configuration_telechat2.py: 4.48kB [00:00, 32.8MB/s] 2025-10-10T01:56:03.8799209Z A new version of the following files was downloaded from https://huggingface.co/Tele-AI/TeleChat2-3B: 2025-10-10T01:56:03.8799767Z - configuration_telechat2.py 2025-10-10T01:56:03.8800392Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:56:11.2608731Z INFO 10-10 01:56:11 [model.py:551] Resolved architecture: TeleChat2ForCausalLM 2025-10-10T01:56:11.2609241Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:56:11.3299899Z ERROR 10-10 01:56:11 [config.py:278] Error retrieving safetensors: 'Tele-AI/TeleChat2-3B' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files., retrying 1 of 2 2025-10-10T01:56:13.3941820Z ERROR 10-10 01:56:13 [config.py:276] Error retrieving safetensors: 'Tele-AI/TeleChat2-3B' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files. 2025-10-10T01:56:13.4184492Z INFO 10-10 01:56:13 [model.py:1765] Downcasting torch.float32 to torch.bfloat16. 2025-10-10T01:56:13.4189768Z INFO 10-10 01:56:13 [model.py:1545] Using max model len 32768 2025-10-10T01:56:13.4192451Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:56:13.4822707Z INFO 10-10 01:56:13 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:56:13.5469022Z 2025-10-10T01:56:13.5471379Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:13.5471750Z tokenizer_config.json: 5.07kB [00:00, 24.3MB/s] 2025-10-10T01:56:13.5867243Z 2025-10-10T01:56:13.5870077Z tokenization_telechat2.py: 0.00B [00:00, ?B/s] 2025-10-10T01:56:13.5870466Z tokenization_telechat2.py: 8.82kB [00:00, 21.1MB/s] 2025-10-10T01:56:13.5922925Z A new version of the following files was downloaded from https://huggingface.co/Tele-AI/TeleChat2-3B: 2025-10-10T01:56:13.5923447Z - tokenization_telechat2.py 2025-10-10T01:56:13.5924068Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:56:13.7261484Z 2025-10-10T01:56:13.9653674Z tokenizer.model: 0% 0.00/2.20M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:14.9464435Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:14.9464896Z 2025-10-10T01:56:14.9465298Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:14.9465697Z 2025-10-10T01:56:14.9466391Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:14.9467008Z 2025-10-10T01:56:14.9467409Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:14.9467790Z 2025-10-10T01:56:14.9468080Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9468361Z 2025-10-10T01:56:14.9468845Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:14.9469289Z 2025-10-10T01:56:14.9469630Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:14.9470893Z 2025-10-10T01:56:14.9471494Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:14.9472031Z 2025-10-10T01:56:14.9472357Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:14.9472678Z 2025-10-10T01:56:14.9472960Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9473243Z 2025-10-10T01:56:14.9473746Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:14.9474217Z 2025-10-10T01:56:14.9474501Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] self._init_executor() 2025-10-10T01:56:14.9475004Z 2025-10-10T01:56:14.9476092Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:14.9476831Z 2025-10-10T01:56:14.9477476Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:14.9478032Z 2025-10-10T01:56:14.9478779Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:14.9479538Z 2025-10-10T01:56:14.9479922Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:14.9480272Z 2025-10-10T01:56:14.9480567Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9480856Z 2025-10-10T01:56:14.9481358Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:14.9481845Z 2025-10-10T01:56:14.9482119Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:14.9482400Z 2025-10-10T01:56:14.9482655Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9482921Z 2025-10-10T01:56:14.9483443Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:14.9483919Z 2025-10-10T01:56:14.9484217Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:14.9484507Z 2025-10-10T01:56:14.9484794Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9485078Z 2025-10-10T01:56:14.9485628Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:14.9486115Z 2025-10-10T01:56:14.9486420Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:14.9486739Z 2025-10-10T01:56:14.9487020Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9487378Z 2025-10-10T01:56:14.9487835Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:14.9488251Z 2025-10-10T01:56:14.9488648Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:14.9489024Z 2025-10-10T01:56:14.9489318Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9489606Z 2025-10-10T01:56:14.9489959Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:14.9490286Z 2025-10-10T01:56:14.9490645Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:14.9490982Z 2025-10-10T01:56:14.9491362Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:14.9491721Z 2025-10-10T01:56:14.9492072Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:14.9492417Z 2025-10-10T01:56:14.9492826Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:14.9493197Z 2025-10-10T01:56:14.9493575Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:14.9493943Z 2025-10-10T01:56:14.9494452Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:14.9494926Z 2025-10-10T01:56:14.9495279Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:14.9495617Z 2025-10-10T01:56:14.9496373Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:14.9496856Z 2025-10-10T01:56:14.9497253Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:14.9497617Z 2025-10-10T01:56:14.9498174Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:14.9498678Z 2025-10-10T01:56:14.9498973Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:14.9499270Z 2025-10-10T01:56:14.9499896Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:14.9500471Z 2025-10-10T01:56:14.9500837Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:14.9501193Z 2025-10-10T01:56:14.9501447Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9501712Z 2025-10-10T01:56:14.9502269Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:14.9502914Z 2025-10-10T01:56:14.9503219Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:14.9503519Z 2025-10-10T01:56:14.9503874Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9504219Z 2025-10-10T01:56:14.9504845Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:14.9505416Z 2025-10-10T01:56:14.9505720Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:14.9506009Z 2025-10-10T01:56:14.9506271Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9506548Z 2025-10-10T01:56:14.9507151Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:14.9507710Z 2025-10-10T01:56:14.9508034Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:14.9508352Z 2025-10-10T01:56:14.9508699Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9508982Z 2025-10-10T01:56:14.9509512Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:14.9509998Z 2025-10-10T01:56:14.9510285Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:14.9510573Z 2025-10-10T01:56:14.9510848Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9511121Z 2025-10-10T01:56:14.9511647Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:14.9512135Z 2025-10-10T01:56:14.9512457Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:14.9512775Z 2025-10-10T01:56:14.9513006Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:14.9513262Z 2025-10-10T01:56:14.9513755Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:14.9514217Z 2025-10-10T01:56:14.9514466Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] raise RuntimeError( 2025-10-10T01:56:14.9514726Z 2025-10-10T01:56:14.9515300Z (EngineCore_DP0 pid=10037) ERROR 10-10 01:56:14 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:14.9516023Z (EngineCore_DP0 pid=10037) Process EngineCore_DP0: 2025-10-10T01:56:14.9516427Z (EngineCore_DP0 pid=10037) Traceback (most recent call last): 2025-10-10T01:56:14.9517038Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:14.9517549Z (EngineCore_DP0 pid=10037) self.run() 2025-10-10T01:56:14.9518081Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:14.9518718Z (EngineCore_DP0 pid=10037) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:14.9519488Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:14.9520097Z (EngineCore_DP0 pid=10037) raise e 2025-10-10T01:56:14.9520768Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:14.9521410Z (EngineCore_DP0 pid=10037) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:14.9521860Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9522485Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:14.9523136Z (EngineCore_DP0 pid=10037) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:14.9523798Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:14.9524426Z (EngineCore_DP0 pid=10037) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:14.9524886Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9525602Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:14.9526207Z (EngineCore_DP0 pid=10037) self._init_executor() 2025-10-10T01:56:14.9526875Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:14.9527577Z (EngineCore_DP0 pid=10037) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:14.9528304Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:14.9529020Z (EngineCore_DP0 pid=10037) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:14.9529524Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9530183Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:14.9530769Z (EngineCore_DP0 pid=10037) return func(*args, **kwargs) 2025-10-10T01:56:14.9531157Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9531798Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:14.9532431Z (EngineCore_DP0 pid=10037) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:14.9532866Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9533559Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:14.9534241Z (EngineCore_DP0 pid=10037) module = importlib.import_module(module_name) 2025-10-10T01:56:14.9534690Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9535281Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:14.9535907Z (EngineCore_DP0 pid=10037) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:14.9536456Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9536980Z (EngineCore_DP0 pid=10037) File "", line 1387, in _gcd_import 2025-10-10T01:56:14.9537694Z (EngineCore_DP0 pid=10037) File "", line 1360, in _find_and_load 2025-10-10T01:56:14.9538448Z (EngineCore_DP0 pid=10037) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:14.9539143Z (EngineCore_DP0 pid=10037) File "", line 935, in _load_unlocked 2025-10-10T01:56:14.9539752Z (EngineCore_DP0 pid=10037) File "", line 999, in exec_module 2025-10-10T01:56:14.9540357Z (EngineCore_DP0 pid=10037) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:14.9541096Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:14.9541780Z (EngineCore_DP0 pid=10037) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:14.9542514Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:14.9543313Z (EngineCore_DP0 pid=10037) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:14.9544134Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:14.9544820Z (EngineCore_DP0 pid=10037) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:14.9545615Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:14.9546466Z (EngineCore_DP0 pid=10037) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:14.9547011Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9547851Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:14.9548693Z (EngineCore_DP0 pid=10037) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:14.9549146Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9549906Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:14.9550618Z (EngineCore_DP0 pid=10037) return _is_fa2_supported(device)[0] 2025-10-10T01:56:14.9551038Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9551758Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:14.9552491Z (EngineCore_DP0 pid=10037) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:14.9552943Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9553615Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:14.9554252Z (EngineCore_DP0 pid=10037) prop = get_device_properties(device) 2025-10-10T01:56:14.9554682Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:14.9555403Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:14.9556104Z (EngineCore_DP0 pid=10037) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:14.9556554Z (EngineCore_DP0 pid=10037) ^^^^^^^^^^^^ 2025-10-10T01:56:14.9557138Z (EngineCore_DP0 pid=10037) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:14.9557706Z (EngineCore_DP0 pid=10037) raise RuntimeError( 2025-10-10T01:56:14.9558397Z (EngineCore_DP0 pid=10037) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:15.3548542Z FAILED 2025-10-10T01:56:15.3677709Z models/test_initialization.py::test_can_initialize_large_subset[MiDashengLMModel] Fork a new process to run a test 10041 2025-10-10T01:56:15.3689140Z Fork a new process to run a test 0 2025-10-10T01:56:15.3966036Z INFO 10-10 01:56:15 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiDashengLMModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mispeech/midashenglm-7b'} 2025-10-10T01:56:15.5169255Z 2025-10-10T01:56:15.5171119Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:15.5171432Z config.json: 1.65kB [00:00, 10.4MB/s] 2025-10-10T01:56:15.5668122Z Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'} 2025-10-10T01:56:15.6659875Z 2025-10-10T01:56:15.6661192Z preprocessor_config.json: 0% 0.00/349 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:24.5618678Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:24.5619139Z 2025-10-10T01:56:24.5619677Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:24.5620064Z 2025-10-10T01:56:24.5620766Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:24.5621401Z 2025-10-10T01:56:24.5621790Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:24.5622174Z 2025-10-10T01:56:24.5622533Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5622891Z 2025-10-10T01:56:24.5623503Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:24.5624066Z 2025-10-10T01:56:24.5624500Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:24.5625016Z 2025-10-10T01:56:24.5625759Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:24.5626601Z 2025-10-10T01:56:24.5626939Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:24.5627257Z 2025-10-10T01:56:24.5627630Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5628012Z 2025-10-10T01:56:24.5628543Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:24.5629020Z 2025-10-10T01:56:24.5629346Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] self._init_executor() 2025-10-10T01:56:24.5629808Z 2025-10-10T01:56:24.5630673Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:24.5631448Z 2025-10-10T01:56:24.5632071Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:24.5632566Z 2025-10-10T01:56:24.5633280Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:24.5633801Z 2025-10-10T01:56:24.5634295Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:24.5634639Z 2025-10-10T01:56:24.5634943Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5635224Z 2025-10-10T01:56:24.5635732Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:24.5636210Z 2025-10-10T01:56:24.5636480Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:24.5636761Z 2025-10-10T01:56:24.5637017Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5637286Z 2025-10-10T01:56:24.5637812Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:24.5638285Z 2025-10-10T01:56:24.5638580Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:24.5638910Z 2025-10-10T01:56:24.5639305Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5639589Z 2025-10-10T01:56:24.5640123Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:24.5640651Z 2025-10-10T01:56:24.5640968Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:24.5641281Z 2025-10-10T01:56:24.5641561Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5641836Z 2025-10-10T01:56:24.5642287Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:24.5642702Z 2025-10-10T01:56:24.5643052Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:24.5643439Z 2025-10-10T01:56:24.5643734Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5644011Z 2025-10-10T01:56:24.5644410Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:24.5644791Z 2025-10-10T01:56:24.5645147Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:24.5645487Z 2025-10-10T01:56:24.5645858Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:24.5646230Z 2025-10-10T01:56:24.5646572Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:24.5646907Z 2025-10-10T01:56:24.5647270Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:24.5647615Z 2025-10-10T01:56:24.5647996Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:24.5648358Z 2025-10-10T01:56:24.5648904Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:24.5649369Z 2025-10-10T01:56:24.5649729Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:24.5650065Z 2025-10-10T01:56:24.5650583Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:24.5651068Z 2025-10-10T01:56:24.5651444Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:24.5651807Z 2025-10-10T01:56:24.5652354Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:24.5652863Z 2025-10-10T01:56:24.5653158Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:24.5653454Z 2025-10-10T01:56:24.5654080Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:24.5654651Z 2025-10-10T01:56:24.5655022Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:24.5655372Z 2025-10-10T01:56:24.5655632Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5655909Z 2025-10-10T01:56:24.5656484Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:24.5657007Z 2025-10-10T01:56:24.5657309Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:24.5657601Z 2025-10-10T01:56:24.5657870Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5658190Z 2025-10-10T01:56:24.5658790Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:24.5659339Z 2025-10-10T01:56:24.5659661Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:24.5659983Z 2025-10-10T01:56:24.5660256Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5660531Z 2025-10-10T01:56:24.5661110Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:24.5661637Z 2025-10-10T01:56:24.5661960Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:24.5662271Z 2025-10-10T01:56:24.5662546Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5662820Z 2025-10-10T01:56:24.5663352Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:24.5663884Z 2025-10-10T01:56:24.5664168Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:24.5664459Z 2025-10-10T01:56:24.5664732Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5665009Z 2025-10-10T01:56:24.5665539Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:24.5666029Z 2025-10-10T01:56:24.5666347Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:24.5666662Z 2025-10-10T01:56:24.5666891Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:24.5667137Z 2025-10-10T01:56:24.5667641Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:24.5668093Z 2025-10-10T01:56:24.5668348Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] raise RuntimeError( 2025-10-10T01:56:24.5668607Z 2025-10-10T01:56:24.5669174Z (EngineCore_DP0 pid=10120) ERROR 10-10 01:56:24 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:24.5669877Z (EngineCore_DP0 pid=10120) Process EngineCore_DP0: 2025-10-10T01:56:24.5670283Z (EngineCore_DP0 pid=10120) Traceback (most recent call last): 2025-10-10T01:56:24.5670896Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:24.5671426Z (EngineCore_DP0 pid=10120) self.run() 2025-10-10T01:56:24.5671957Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:24.5672517Z (EngineCore_DP0 pid=10120) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:24.5673188Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:24.5673789Z (EngineCore_DP0 pid=10120) raise e 2025-10-10T01:56:24.5674381Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:24.5675073Z (EngineCore_DP0 pid=10120) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:24.5675586Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5676226Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:24.5676877Z (EngineCore_DP0 pid=10120) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:24.5677552Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:24.5678177Z (EngineCore_DP0 pid=10120) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:24.5678627Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5679344Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:24.5679938Z (EngineCore_DP0 pid=10120) self._init_executor() 2025-10-10T01:56:24.5680669Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:24.5681367Z (EngineCore_DP0 pid=10120) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:24.5682084Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:24.5682810Z (EngineCore_DP0 pid=10120) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:24.5683316Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5683960Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:24.5684550Z (EngineCore_DP0 pid=10120) return func(*args, **kwargs) 2025-10-10T01:56:24.5684933Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5685567Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:24.5686197Z (EngineCore_DP0 pid=10120) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:24.5686635Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5687310Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:24.5687966Z (EngineCore_DP0 pid=10120) module = importlib.import_module(module_name) 2025-10-10T01:56:24.5688409Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5688995Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:24.5689617Z (EngineCore_DP0 pid=10120) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:24.5690107Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5690605Z (EngineCore_DP0 pid=10120) File "", line 1387, in _gcd_import 2025-10-10T01:56:24.5691158Z (EngineCore_DP0 pid=10120) File "", line 1360, in _find_and_load 2025-10-10T01:56:24.5691803Z (EngineCore_DP0 pid=10120) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:24.5692422Z (EngineCore_DP0 pid=10120) File "", line 935, in _load_unlocked 2025-10-10T01:56:24.5693034Z (EngineCore_DP0 pid=10120) File "", line 999, in exec_module 2025-10-10T01:56:24.5693641Z (EngineCore_DP0 pid=10120) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:24.5694385Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:24.5695062Z (EngineCore_DP0 pid=10120) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:24.5695780Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:24.5696704Z (EngineCore_DP0 pid=10120) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:24.5705284Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:24.5706165Z (EngineCore_DP0 pid=10120) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:24.5707030Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:24.5707863Z (EngineCore_DP0 pid=10120) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:24.5708357Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5709066Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:24.5709779Z (EngineCore_DP0 pid=10120) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:24.5710231Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5710989Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:24.5711710Z (EngineCore_DP0 pid=10120) return _is_fa2_supported(device)[0] 2025-10-10T01:56:24.5712137Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5712860Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:24.5713582Z (EngineCore_DP0 pid=10120) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:24.5714038Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5714729Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:24.5715382Z (EngineCore_DP0 pid=10120) prop = get_device_properties(device) 2025-10-10T01:56:24.5715815Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:24.5716487Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:24.5717158Z (EngineCore_DP0 pid=10120) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:24.5717654Z (EngineCore_DP0 pid=10120) ^^^^^^^^^^^^ 2025-10-10T01:56:24.5718241Z (EngineCore_DP0 pid=10120) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:24.5718906Z (EngineCore_DP0 pid=10120) raise RuntimeError( 2025-10-10T01:56:24.5719771Z (EngineCore_DP0 pid=10120) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:24.9839764Z FAILED 2025-10-10T01:56:24.9969340Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2Model] Fork a new process to run a test 10124 2025-10-10T01:56:24.9981431Z Fork a new process to run a test 0 2025-10-10T01:56:25.0257487Z INFO 10-10 01:56:25 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2Model', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ssmits/Qwen2-7B-Instruct-embed-base'} 2025-10-10T01:56:25.1880985Z 2025-10-10T01:56:25.1882319Z config.json: 0% 0.00/706 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:26.4407783Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:26.4408455Z 2025-10-10T01:56:26.4409090Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:26.4409571Z 2025-10-10T01:56:26.4410261Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:26.4410876Z 2025-10-10T01:56:26.4411515Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:26.4411908Z 2025-10-10T01:56:26.4412262Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4412610Z 2025-10-10T01:56:26.4413211Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:26.4414011Z 2025-10-10T01:56:26.4414755Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:26.4415326Z 2025-10-10T01:56:26.4415951Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:26.4416511Z 2025-10-10T01:56:26.4416905Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:26.4417288Z 2025-10-10T01:56:26.4417889Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4418451Z 2025-10-10T01:56:26.4419351Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:26.4420186Z 2025-10-10T01:56:26.4420631Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] self._init_executor() 2025-10-10T01:56:26.4421093Z 2025-10-10T01:56:26.4422049Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:26.4422784Z 2025-10-10T01:56:26.4423125Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:26.4423447Z 2025-10-10T01:56:26.4423993Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:26.4424492Z 2025-10-10T01:56:26.4424993Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:26.4425334Z 2025-10-10T01:56:26.4425713Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4426084Z 2025-10-10T01:56:26.4426600Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:26.4427085Z 2025-10-10T01:56:26.4427370Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:26.4427662Z 2025-10-10T01:56:26.4427916Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4428180Z 2025-10-10T01:56:26.4428710Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:26.4429204Z 2025-10-10T01:56:26.4429500Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:26.4429792Z 2025-10-10T01:56:26.4430067Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4430379Z 2025-10-10T01:56:26.4430943Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:26.4431435Z 2025-10-10T01:56:26.4431745Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:26.4432046Z 2025-10-10T01:56:26.4432325Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4432606Z 2025-10-10T01:56:26.4433066Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:26.4433493Z 2025-10-10T01:56:26.4433872Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:26.4434202Z 2025-10-10T01:56:26.4434494Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4434770Z 2025-10-10T01:56:26.4435111Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:26.4435449Z 2025-10-10T01:56:26.4435799Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:26.4436149Z 2025-10-10T01:56:26.4436525Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:26.4436894Z 2025-10-10T01:56:26.4437240Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:26.4437594Z 2025-10-10T01:56:26.4437966Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:26.4438317Z 2025-10-10T01:56:26.4438701Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:26.4439196Z 2025-10-10T01:56:26.4439787Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:26.4440275Z 2025-10-10T01:56:26.4440677Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:26.4441049Z 2025-10-10T01:56:26.4441593Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:26.4442084Z 2025-10-10T01:56:26.4442457Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:26.4442823Z 2025-10-10T01:56:26.4443376Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:26.4443893Z 2025-10-10T01:56:26.4444189Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:26.4444493Z 2025-10-10T01:56:26.4445161Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:26.4445741Z 2025-10-10T01:56:26.4446109Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:26.4446459Z 2025-10-10T01:56:26.4446724Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4446988Z 2025-10-10T01:56:26.4447557Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:26.4448085Z 2025-10-10T01:56:26.4448392Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:26.4448687Z 2025-10-10T01:56:26.4448963Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4449243Z 2025-10-10T01:56:26.4449844Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:26.4450403Z 2025-10-10T01:56:26.4450692Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:26.4450982Z 2025-10-10T01:56:26.4451245Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4451529Z 2025-10-10T01:56:26.4452123Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:26.4452693Z 2025-10-10T01:56:26.4453020Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:26.4453347Z 2025-10-10T01:56:26.4453629Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4453906Z 2025-10-10T01:56:26.4454446Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:26.4454982Z 2025-10-10T01:56:26.4455270Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:26.4455556Z 2025-10-10T01:56:26.4455864Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4456142Z 2025-10-10T01:56:26.4456716Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:26.4457208Z 2025-10-10T01:56:26.4457527Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:26.4457836Z 2025-10-10T01:56:26.4458062Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:26.4458309Z 2025-10-10T01:56:26.4458786Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:26.4459238Z 2025-10-10T01:56:26.4459484Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] raise RuntimeError( 2025-10-10T01:56:26.4459743Z 2025-10-10T01:56:26.4460351Z (EngineCore_DP0 pid=10132) ERROR 10-10 01:56:26 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:26.4461049Z (EngineCore_DP0 pid=10132) Process EngineCore_DP0: 2025-10-10T01:56:26.4461442Z (EngineCore_DP0 pid=10132) Traceback (most recent call last): 2025-10-10T01:56:26.4462065Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:26.4462578Z (EngineCore_DP0 pid=10132) self.run() 2025-10-10T01:56:26.4463107Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:26.4463667Z (EngineCore_DP0 pid=10132) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:26.4464329Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:26.4464885Z (EngineCore_DP0 pid=10132) raise e 2025-10-10T01:56:26.4465470Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:26.4466101Z (EngineCore_DP0 pid=10132) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:26.4466546Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4467163Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:26.4467815Z (EngineCore_DP0 pid=10132) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:26.4468477Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:26.4469100Z (EngineCore_DP0 pid=10132) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:26.4469554Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4470200Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:26.4470796Z (EngineCore_DP0 pid=10132) self._init_executor() 2025-10-10T01:56:26.4471471Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:26.4472210Z (EngineCore_DP0 pid=10132) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:26.4472991Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:26.4473745Z (EngineCore_DP0 pid=10132) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:26.4474238Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4474881Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:26.4475470Z (EngineCore_DP0 pid=10132) return func(*args, **kwargs) 2025-10-10T01:56:26.4475863Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4476506Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:26.4477129Z (EngineCore_DP0 pid=10132) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:26.4477559Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4478269Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:26.4478933Z (EngineCore_DP0 pid=10132) module = importlib.import_module(module_name) 2025-10-10T01:56:26.4479450Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4480033Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:26.4480677Z (EngineCore_DP0 pid=10132) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:26.4481174Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4481687Z (EngineCore_DP0 pid=10132) File "", line 1387, in _gcd_import 2025-10-10T01:56:26.4482250Z (EngineCore_DP0 pid=10132) File "", line 1360, in _find_and_load 2025-10-10T01:56:26.4482829Z (EngineCore_DP0 pid=10132) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:26.4483408Z (EngineCore_DP0 pid=10132) File "", line 935, in _load_unlocked 2025-10-10T01:56:26.4483982Z (EngineCore_DP0 pid=10132) File "", line 999, in exec_module 2025-10-10T01:56:26.4484583Z (EngineCore_DP0 pid=10132) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:26.4485308Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:26.4485977Z (EngineCore_DP0 pid=10132) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:26.4486697Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:26.4487426Z (EngineCore_DP0 pid=10132) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:26.4488205Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:26.4488874Z (EngineCore_DP0 pid=10132) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:26.4489717Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:26.4490563Z (EngineCore_DP0 pid=10132) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:26.4491083Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4491776Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:26.4492455Z (EngineCore_DP0 pid=10132) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:26.4492908Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4493655Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:26.4494354Z (EngineCore_DP0 pid=10132) return _is_fa2_supported(device)[0] 2025-10-10T01:56:26.4494766Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4495533Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:26.4496452Z (EngineCore_DP0 pid=10132) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:26.4496920Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4497604Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:26.4498256Z (EngineCore_DP0 pid=10132) prop = get_device_properties(device) 2025-10-10T01:56:26.4498673Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:26.4499332Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:26.4499999Z (EngineCore_DP0 pid=10132) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:26.4500404Z (EngineCore_DP0 pid=10132) ^^^^^^^^^^^^ 2025-10-10T01:56:26.4500994Z (EngineCore_DP0 pid=10132) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:26.4501561Z (EngineCore_DP0 pid=10132) raise RuntimeError( 2025-10-10T01:56:26.4502251Z (EngineCore_DP0 pid=10132) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:26.8524848Z FAILED 2025-10-10T01:56:26.8653451Z models/test_initialization.py::test_can_initialize_large_subset[BertForTokenClassification] Fork a new process to run a test 10136 2025-10-10T01:56:26.8665331Z Fork a new process to run a test 0 2025-10-10T01:56:26.8938835Z INFO 10-10 01:56:26 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BertForTokenClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'boltuix/NeuroBERT-NER'} 2025-10-10T01:56:27.0320475Z 2025-10-10T01:56:27.0322326Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:27.0322669Z config.json: 1.84kB [00:00, 10.8MB/s] 2025-10-10T01:56:33.7699950Z INFO 10-10 01:56:33 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:56:33.7701133Z INFO 10-10 01:56:33 [model.py:551] Resolved architecture: BertForTokenClassification 2025-10-10T01:56:33.7701594Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:56:33.7948392Z INFO 10-10 01:56:33 [model.py:1765] Downcasting torch.float32 to torch.float16. 2025-10-10T01:56:33.8334322Z 2025-10-10T01:56:33.8335339Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:33.8335819Z tokenizer_config.json: 1.30kB [00:00, 13.3MB/s] 2025-10-10T01:56:33.8344616Z INFO 10-10 01:56:33 [model.py:1545] Using max model len 512 2025-10-10T01:56:33.8695505Z INFO 10-10 01:56:33 [arg_utils.py:1580] (Disabling) chunked prefill by default 2025-10-10T01:56:33.8696374Z INFO 10-10 01:56:33 [arg_utils.py:1583] (Disabling) prefix caching by default 2025-10-10T01:56:34.0126079Z INFO 10-10 01:56:34 [vllm.py:404] Only "last" pooling supports chunked prefill and prefix caching; disabling both. 2025-10-10T01:56:34.1292062Z 2025-10-10T01:56:34.1313698Z vocab.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:56:34.1314040Z vocab.txt: 232kB [00:00, 107MB/s] 2025-10-10T01:56:34.1919488Z 2025-10-10T01:56:34.1955139Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:34.1955445Z tokenizer.json: 711kB [00:00, 202MB/s] 2025-10-10T01:56:34.2697776Z 2025-10-10T01:56:34.2698638Z special_tokens_map.json: 0% 0.00/695 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:34.6076796Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:34.6077687Z 2025-10-10T01:56:34.6078159Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:34.6078756Z 2025-10-10T01:56:34.6079753Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:34.6080492Z 2025-10-10T01:56:34.6080905Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:34.6081289Z 2025-10-10T01:56:34.6081641Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6081999Z 2025-10-10T01:56:34.6082605Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:34.6083167Z 2025-10-10T01:56:34.6083597Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:34.6084109Z 2025-10-10T01:56:34.6085000Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:34.6085562Z 2025-10-10T01:56:34.6086001Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:34.6086344Z 2025-10-10T01:56:34.6086637Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6086918Z 2025-10-10T01:56:34.6087528Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:34.6088410Z 2025-10-10T01:56:34.6088865Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] self._init_executor() 2025-10-10T01:56:34.6089308Z 2025-10-10T01:56:34.6090213Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:34.6091141Z 2025-10-10T01:56:34.6091640Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:34.6092000Z 2025-10-10T01:56:34.6092573Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:34.6093082Z 2025-10-10T01:56:34.6093429Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:34.6093775Z 2025-10-10T01:56:34.6094068Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6094355Z 2025-10-10T01:56:34.6094855Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:34.6095317Z 2025-10-10T01:56:34.6095589Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:34.6095859Z 2025-10-10T01:56:34.6096298Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6096568Z 2025-10-10T01:56:34.6097109Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:34.6097708Z 2025-10-10T01:56:34.6098002Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:34.6098442Z 2025-10-10T01:56:34.6098787Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6099082Z 2025-10-10T01:56:34.6099637Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:34.6100136Z 2025-10-10T01:56:34.6100443Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:34.6100749Z 2025-10-10T01:56:34.6101024Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6101300Z 2025-10-10T01:56:34.6101747Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:34.6102162Z 2025-10-10T01:56:34.6102514Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:34.6102921Z 2025-10-10T01:56:34.6103225Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6103509Z 2025-10-10T01:56:34.6103853Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:34.6104189Z 2025-10-10T01:56:34.6104542Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:34.6104934Z 2025-10-10T01:56:34.6105384Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:34.6105821Z 2025-10-10T01:56:34.6106226Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:34.6106624Z 2025-10-10T01:56:34.6106994Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:34.6107344Z 2025-10-10T01:56:34.6107724Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:34.6108083Z 2025-10-10T01:56:34.6108595Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:34.6109055Z 2025-10-10T01:56:34.6109411Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:34.6109747Z 2025-10-10T01:56:34.6110267Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:34.6110749Z 2025-10-10T01:56:34.6111120Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:34.6111487Z 2025-10-10T01:56:34.6112041Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:34.6112623Z 2025-10-10T01:56:34.6112924Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:34.6113225Z 2025-10-10T01:56:34.6113899Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:34.6114552Z 2025-10-10T01:56:34.6114924Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:34.6115283Z 2025-10-10T01:56:34.6115542Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6115804Z 2025-10-10T01:56:34.6116369Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:34.6116885Z 2025-10-10T01:56:34.6117185Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:34.6117480Z 2025-10-10T01:56:34.6117752Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6118032Z 2025-10-10T01:56:34.6118676Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:34.6119350Z 2025-10-10T01:56:34.6119637Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:34.6119924Z 2025-10-10T01:56:34.6120183Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6120455Z 2025-10-10T01:56:34.6121031Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:34.6121555Z 2025-10-10T01:56:34.6121877Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:34.6122190Z 2025-10-10T01:56:34.6122480Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6122754Z 2025-10-10T01:56:34.6123281Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:34.6123764Z 2025-10-10T01:56:34.6124046Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:34.6124338Z 2025-10-10T01:56:34.6124604Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6124883Z 2025-10-10T01:56:34.6125414Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:34.6125906Z 2025-10-10T01:56:34.6126222Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:34.6126543Z 2025-10-10T01:56:34.6126768Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:34.6127019Z 2025-10-10T01:56:34.6127512Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:34.6128034Z 2025-10-10T01:56:34.6128286Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] raise RuntimeError( 2025-10-10T01:56:34.6128543Z 2025-10-10T01:56:34.6129164Z (EngineCore_DP0 pid=10195) ERROR 10-10 01:56:34 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:34.6129908Z (EngineCore_DP0 pid=10195) Process EngineCore_DP0: 2025-10-10T01:56:34.6130336Z (EngineCore_DP0 pid=10195) Traceback (most recent call last): 2025-10-10T01:56:34.6130949Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:34.6131464Z (EngineCore_DP0 pid=10195) self.run() 2025-10-10T01:56:34.6131994Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:34.6132559Z (EngineCore_DP0 pid=10195) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:34.6133235Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:34.6133791Z (EngineCore_DP0 pid=10195) raise e 2025-10-10T01:56:34.6134415Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:34.6135074Z (EngineCore_DP0 pid=10195) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:34.6135515Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6136137Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:34.6136779Z (EngineCore_DP0 pid=10195) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:34.6137433Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:34.6138054Z (EngineCore_DP0 pid=10195) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:34.6138534Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6139185Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:34.6139778Z (EngineCore_DP0 pid=10195) self._init_executor() 2025-10-10T01:56:34.6140440Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:34.6141122Z (EngineCore_DP0 pid=10195) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:34.6141847Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:34.6142560Z (EngineCore_DP0 pid=10195) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:34.6143059Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6143700Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:34.6144280Z (EngineCore_DP0 pid=10195) return func(*args, **kwargs) 2025-10-10T01:56:34.6144680Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6145387Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:34.6146056Z (EngineCore_DP0 pid=10195) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:34.6146534Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6147196Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:34.6147855Z (EngineCore_DP0 pid=10195) module = importlib.import_module(module_name) 2025-10-10T01:56:34.6148313Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6148898Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:34.6149526Z (EngineCore_DP0 pid=10195) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:34.6150010Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6150507Z (EngineCore_DP0 pid=10195) File "", line 1387, in _gcd_import 2025-10-10T01:56:34.6151112Z (EngineCore_DP0 pid=10195) File "", line 1360, in _find_and_load 2025-10-10T01:56:34.6151710Z (EngineCore_DP0 pid=10195) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:34.6152291Z (EngineCore_DP0 pid=10195) File "", line 935, in _load_unlocked 2025-10-10T01:56:34.6152856Z (EngineCore_DP0 pid=10195) File "", line 999, in exec_module 2025-10-10T01:56:34.6153472Z (EngineCore_DP0 pid=10195) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:34.6154207Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:34.6154885Z (EngineCore_DP0 pid=10195) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:34.6155607Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:34.6156328Z (EngineCore_DP0 pid=10195) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:34.6157104Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:34.6157779Z (EngineCore_DP0 pid=10195) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:34.6158573Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:34.6159465Z (EngineCore_DP0 pid=10195) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:34.6159952Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6160642Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:34.6161340Z (EngineCore_DP0 pid=10195) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:34.6161781Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6162525Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:34.6163287Z (EngineCore_DP0 pid=10195) return _is_fa2_supported(device)[0] 2025-10-10T01:56:34.6163747Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6164501Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:34.6165236Z (EngineCore_DP0 pid=10195) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:34.6165693Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6166360Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:34.6166993Z (EngineCore_DP0 pid=10195) prop = get_device_properties(device) 2025-10-10T01:56:34.6167409Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:34.6168078Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:34.6168744Z (EngineCore_DP0 pid=10195) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:34.6169187Z (EngineCore_DP0 pid=10195) ^^^^^^^^^^^^ 2025-10-10T01:56:34.6169774Z (EngineCore_DP0 pid=10195) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:34.6170342Z (EngineCore_DP0 pid=10195) raise RuntimeError( 2025-10-10T01:56:34.6171033Z (EngineCore_DP0 pid=10195) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:35.0129290Z FAILED 2025-10-10T01:56:35.0258102Z models/test_initialization.py::test_can_initialize_large_subset[InternLM3ForCausalLM] Fork a new process to run a test 10199 2025-10-10T01:56:35.0270276Z Fork a new process to run a test 0 2025-10-10T01:56:35.0547310Z INFO 10-10 01:56:35 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternLM3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'internlm/internlm3-8b-instruct'} 2025-10-10T01:56:35.1367668Z 2025-10-10T01:56:35.1369684Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:35.1369978Z config.json: 1.01kB [00:00, 5.19MB/s] 2025-10-10T01:56:35.2089932Z 2025-10-10T01:56:35.2091783Z configuration_internlm3.py: 0.00B [00:00, ?B/s] 2025-10-10T01:56:35.2092180Z configuration_internlm3.py: 10.5kB [00:00, 60.1MB/s] 2025-10-10T01:56:35.2176967Z A new version of the following files was downloaded from https://huggingface.co/internlm/internlm3-8b-instruct: 2025-10-10T01:56:35.2177529Z - configuration_internlm3.py 2025-10-10T01:56:35.2178157Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:56:35.4264981Z INFO 10-10 01:56:35 [model.py:551] Resolved architecture: InternLM3ForCausalLM 2025-10-10T01:56:35.4265481Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:56:35.4512900Z INFO 10-10 01:56:35 [model.py:1545] Using max model len 196608 2025-10-10T01:56:35.4514859Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:56:35.5083867Z INFO 10-10 01:56:35 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:56:35.5590953Z 2025-10-10T01:56:35.5593054Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:56:35.5593441Z tokenizer_config.json: 5.69kB [00:00, 39.1MB/s] 2025-10-10T01:56:35.5948853Z 2025-10-10T01:56:35.5952762Z tokenization_internlm3.py: 0.00B [00:00, ?B/s] 2025-10-10T01:56:35.5953156Z tokenization_internlm3.py: 13.2kB [00:00, 37.1MB/s] 2025-10-10T01:56:35.6009597Z A new version of the following files was downloaded from https://huggingface.co/internlm/internlm3-8b-instruct: 2025-10-10T01:56:35.6010302Z - tokenization_internlm3.py 2025-10-10T01:56:35.6010925Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:56:35.7161077Z 2025-10-10T01:56:35.9698595Z tokenizer.model: 0% 0.00/2.48M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:36.8417398Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:36.8417801Z 2025-10-10T01:56:36.8418424Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:36.8418973Z 2025-10-10T01:56:36.8419812Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:36.8420527Z 2025-10-10T01:56:36.8420947Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:36.8421333Z 2025-10-10T01:56:36.8421682Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8422038Z 2025-10-10T01:56:36.8422643Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:36.8423201Z 2025-10-10T01:56:36.8423637Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:36.8424049Z 2025-10-10T01:56:36.8424652Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:36.8425207Z 2025-10-10T01:56:36.8425720Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:36.8426120Z 2025-10-10T01:56:36.8426474Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8426824Z 2025-10-10T01:56:36.8427464Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:36.8427985Z 2025-10-10T01:56:36.8428244Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] self._init_executor() 2025-10-10T01:56:36.8428510Z 2025-10-10T01:56:36.8429076Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:36.8429575Z 2025-10-10T01:56:36.8429908Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:36.8430232Z 2025-10-10T01:56:36.8430781Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:36.8431280Z 2025-10-10T01:56:36.8431623Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:36.8431971Z 2025-10-10T01:56:36.8432263Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8432550Z 2025-10-10T01:56:36.8433039Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:36.8433499Z 2025-10-10T01:56:36.8433768Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:36.8434042Z 2025-10-10T01:56:36.8434296Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8434554Z 2025-10-10T01:56:36.8435074Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:36.8435607Z 2025-10-10T01:56:36.8435904Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:36.8436192Z 2025-10-10T01:56:36.8436530Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8436837Z 2025-10-10T01:56:36.8437372Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:36.8437868Z 2025-10-10T01:56:36.8438171Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:36.8438475Z 2025-10-10T01:56:36.8438748Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8439030Z 2025-10-10T01:56:36.8439599Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:36.8440022Z 2025-10-10T01:56:36.8440375Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:36.8440713Z 2025-10-10T01:56:36.8441067Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8441351Z 2025-10-10T01:56:36.8441702Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:36.8442032Z 2025-10-10T01:56:36.8442384Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:36.8442727Z 2025-10-10T01:56:36.8443100Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:36.8443464Z 2025-10-10T01:56:36.8443809Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:36.8444148Z 2025-10-10T01:56:36.8444510Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:36.8444864Z 2025-10-10T01:56:36.8445237Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:36.8445597Z 2025-10-10T01:56:36.8446109Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:36.8446578Z 2025-10-10T01:56:36.8446934Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:36.8447274Z 2025-10-10T01:56:36.8447805Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:36.8448287Z 2025-10-10T01:56:36.8448660Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:36.8449015Z 2025-10-10T01:56:36.8449562Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:36.8450072Z 2025-10-10T01:56:36.8450429Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:36.8450732Z 2025-10-10T01:56:36.8451404Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:36.8452016Z 2025-10-10T01:56:36.8452384Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:36.8452745Z 2025-10-10T01:56:36.8452999Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8453258Z 2025-10-10T01:56:36.8453832Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:36.8454352Z 2025-10-10T01:56:36.8454653Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:36.8454944Z 2025-10-10T01:56:36.8455222Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8455496Z 2025-10-10T01:56:36.8456134Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:36.8456683Z 2025-10-10T01:56:36.8456965Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:36.8457258Z 2025-10-10T01:56:36.8457516Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8457788Z 2025-10-10T01:56:36.8458363Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:36.8458891Z 2025-10-10T01:56:36.8459209Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:36.8459528Z 2025-10-10T01:56:36.8459802Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8460076Z 2025-10-10T01:56:36.8460604Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:36.8461086Z 2025-10-10T01:56:36.8461379Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:36.8461669Z 2025-10-10T01:56:36.8461943Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8462216Z 2025-10-10T01:56:36.8462746Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:36.8463243Z 2025-10-10T01:56:36.8463569Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:36.8463881Z 2025-10-10T01:56:36.8464110Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:36.8464357Z 2025-10-10T01:56:36.8464834Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:36.8465340Z 2025-10-10T01:56:36.8465601Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] raise RuntimeError( 2025-10-10T01:56:36.8465859Z 2025-10-10T01:56:36.8466506Z (EngineCore_DP0 pid=10227) ERROR 10-10 01:56:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:36.8467261Z (EngineCore_DP0 pid=10227) Process EngineCore_DP0: 2025-10-10T01:56:36.8467662Z (EngineCore_DP0 pid=10227) Traceback (most recent call last): 2025-10-10T01:56:36.8468288Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:36.8468806Z (EngineCore_DP0 pid=10227) self.run() 2025-10-10T01:56:36.8469334Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:36.8469904Z (EngineCore_DP0 pid=10227) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:36.8470580Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:36.8471123Z (EngineCore_DP0 pid=10227) raise e 2025-10-10T01:56:36.8471769Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:36.8472413Z (EngineCore_DP0 pid=10227) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:36.8472873Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8473495Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:36.8474138Z (EngineCore_DP0 pid=10227) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:36.8474790Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:36.8475416Z (EngineCore_DP0 pid=10227) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:36.8475882Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8476534Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:36.8477120Z (EngineCore_DP0 pid=10227) self._init_executor() 2025-10-10T01:56:36.8477779Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:36.8478465Z (EngineCore_DP0 pid=10227) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:36.8479251Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:36.8479972Z (EngineCore_DP0 pid=10227) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:36.8480538Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8481181Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:36.8481766Z (EngineCore_DP0 pid=10227) return func(*args, **kwargs) 2025-10-10T01:56:36.8482155Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8482794Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:36.8483498Z (EngineCore_DP0 pid=10227) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:36.8483961Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8484666Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:36.8485336Z (EngineCore_DP0 pid=10227) module = importlib.import_module(module_name) 2025-10-10T01:56:36.8485782Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8486362Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:36.8486976Z (EngineCore_DP0 pid=10227) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:36.8487492Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8487994Z (EngineCore_DP0 pid=10227) File "", line 1387, in _gcd_import 2025-10-10T01:56:36.8488554Z (EngineCore_DP0 pid=10227) File "", line 1360, in _find_and_load 2025-10-10T01:56:36.8489185Z (EngineCore_DP0 pid=10227) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:36.8489768Z (EngineCore_DP0 pid=10227) File "", line 935, in _load_unlocked 2025-10-10T01:56:36.8490334Z (EngineCore_DP0 pid=10227) File "", line 999, in exec_module 2025-10-10T01:56:36.8490935Z (EngineCore_DP0 pid=10227) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:36.8491687Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:36.8492364Z (EngineCore_DP0 pid=10227) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:36.8493078Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:36.8493800Z (EngineCore_DP0 pid=10227) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:36.8494584Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:36.8495259Z (EngineCore_DP0 pid=10227) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:36.8496058Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:36.8497260Z (EngineCore_DP0 pid=10227) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:36.8497751Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8498469Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:36.8499155Z (EngineCore_DP0 pid=10227) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:36.8499592Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8500338Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:36.8501173Z (EngineCore_DP0 pid=10227) return _is_fa2_supported(device)[0] 2025-10-10T01:56:36.8501592Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8502380Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:36.8503181Z (EngineCore_DP0 pid=10227) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:36.8503645Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8504323Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:36.8504957Z (EngineCore_DP0 pid=10227) prop = get_device_properties(device) 2025-10-10T01:56:36.8505378Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:36.8506054Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:36.8506719Z (EngineCore_DP0 pid=10227) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:36.8507117Z (EngineCore_DP0 pid=10227) ^^^^^^^^^^^^ 2025-10-10T01:56:36.8507760Z (EngineCore_DP0 pid=10227) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:36.8508336Z (EngineCore_DP0 pid=10227) raise RuntimeError( 2025-10-10T01:56:36.8509027Z (EngineCore_DP0 pid=10227) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:37.2504120Z FAILED 2025-10-10T01:56:37.2632924Z models/test_initialization.py::test_can_initialize_large_subset[EagleLlamaForCausalLM] Fork a new process to run a test 10231 2025-10-10T01:56:37.2645030Z Fork a new process to run a test 0 2025-10-10T01:56:37.2922539Z INFO 10-10 01:56:37 [utils.py:233] non-default args: {'tokenizer': 'meta-llama/Meta-Llama-3-8B-Instruct', 'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='EagleLlamaForCausalLM', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'yuhuili/EAGLE-LLaMA3-Instruct-8B', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'meta-llama/Meta-Llama-3-8B-Instruct'} 2025-10-10T01:56:37.4649253Z 2025-10-10T01:56:37.4650210Z config.json: 0% 0.00/654 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:39.0676051Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:39.0676531Z 2025-10-10T01:56:39.0677141Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:39.0677535Z 2025-10-10T01:56:39.0678232Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:39.0678828Z 2025-10-10T01:56:39.0679367Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:39.0679716Z 2025-10-10T01:56:39.0680278Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0680568Z 2025-10-10T01:56:39.0681141Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:39.0681677Z 2025-10-10T01:56:39.0682231Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:39.0682763Z 2025-10-10T01:56:39.0683280Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:39.0683731Z 2025-10-10T01:56:39.0684047Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:39.0684359Z 2025-10-10T01:56:39.0684648Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0684924Z 2025-10-10T01:56:39.0685463Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:39.0686232Z 2025-10-10T01:56:39.0686900Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] self._init_executor() 2025-10-10T01:56:39.0687227Z 2025-10-10T01:56:39.0688079Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:39.0689020Z 2025-10-10T01:56:39.0689659Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:39.0690032Z 2025-10-10T01:56:39.0690618Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:39.0691132Z 2025-10-10T01:56:39.0691483Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:39.0691828Z 2025-10-10T01:56:39.0692124Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0692422Z 2025-10-10T01:56:39.0692919Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:39.0693375Z 2025-10-10T01:56:39.0693656Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:39.0693940Z 2025-10-10T01:56:39.0694198Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0694461Z 2025-10-10T01:56:39.0694998Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:39.0695476Z 2025-10-10T01:56:39.0695771Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:39.0696283Z 2025-10-10T01:56:39.0696709Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0696993Z 2025-10-10T01:56:39.0697533Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:39.0698166Z 2025-10-10T01:56:39.0698480Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:39.0698788Z 2025-10-10T01:56:39.0699141Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0699491Z 2025-10-10T01:56:39.0699982Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:39.0700410Z 2025-10-10T01:56:39.0700764Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:39.0701099Z 2025-10-10T01:56:39.0701396Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0701681Z 2025-10-10T01:56:39.0702025Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:39.0702377Z 2025-10-10T01:56:39.0702736Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:39.0703083Z 2025-10-10T01:56:39.0703534Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:39.0703909Z 2025-10-10T01:56:39.0704263Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:39.0704599Z 2025-10-10T01:56:39.0704966Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:39.0705324Z 2025-10-10T01:56:39.0705704Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:39.0706066Z 2025-10-10T01:56:39.0706576Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:39.0707048Z 2025-10-10T01:56:39.0707412Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:39.0707764Z 2025-10-10T01:56:39.0708288Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:39.0708774Z 2025-10-10T01:56:39.0709152Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:39.0709527Z 2025-10-10T01:56:39.0710082Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:39.0710603Z 2025-10-10T01:56:39.0710900Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:39.0711208Z 2025-10-10T01:56:39.0711839Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:39.0712415Z 2025-10-10T01:56:39.0712787Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:39.0713202Z 2025-10-10T01:56:39.0713469Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0713735Z 2025-10-10T01:56:39.0714347Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:39.0714926Z 2025-10-10T01:56:39.0715242Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:39.0715550Z 2025-10-10T01:56:39.0715829Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0716110Z 2025-10-10T01:56:39.0716713Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:39.0717275Z 2025-10-10T01:56:39.0717562Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:39.0717856Z 2025-10-10T01:56:39.0718120Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0718391Z 2025-10-10T01:56:39.0719023Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:39.0719662Z 2025-10-10T01:56:39.0719998Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:39.0720316Z 2025-10-10T01:56:39.0720600Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0720881Z 2025-10-10T01:56:39.0721416Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:39.0721908Z 2025-10-10T01:56:39.0722194Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:39.0722507Z 2025-10-10T01:56:39.0722782Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0723071Z 2025-10-10T01:56:39.0723600Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:39.0724097Z 2025-10-10T01:56:39.0724413Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:39.0724730Z 2025-10-10T01:56:39.0724960Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:39.0725209Z 2025-10-10T01:56:39.0725709Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:39.0726165Z 2025-10-10T01:56:39.0726425Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] raise RuntimeError( 2025-10-10T01:56:39.0726688Z 2025-10-10T01:56:39.0727262Z (EngineCore_DP0 pid=10239) ERROR 10-10 01:56:39 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:39.0727986Z (EngineCore_DP0 pid=10239) Process EngineCore_DP0: 2025-10-10T01:56:39.0728384Z (EngineCore_DP0 pid=10239) Traceback (most recent call last): 2025-10-10T01:56:39.0729069Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:39.0729603Z (EngineCore_DP0 pid=10239) self.run() 2025-10-10T01:56:39.0730174Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:39.0730833Z (EngineCore_DP0 pid=10239) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:39.0731536Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:39.0732099Z (EngineCore_DP0 pid=10239) raise e 2025-10-10T01:56:39.0732697Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:39.0733342Z (EngineCore_DP0 pid=10239) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:39.0733796Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0734431Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:39.0735141Z (EngineCore_DP0 pid=10239) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:39.0735820Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:39.0736452Z (EngineCore_DP0 pid=10239) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:39.0736919Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0737575Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:39.0738173Z (EngineCore_DP0 pid=10239) self._init_executor() 2025-10-10T01:56:39.0738845Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:39.0739671Z (EngineCore_DP0 pid=10239) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:39.0740547Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:39.0741401Z (EngineCore_DP0 pid=10239) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:39.0741916Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0742584Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:39.0743180Z (EngineCore_DP0 pid=10239) return func(*args, **kwargs) 2025-10-10T01:56:39.0743576Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0744230Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:39.0744869Z (EngineCore_DP0 pid=10239) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:39.0745308Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0745987Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:39.0746652Z (EngineCore_DP0 pid=10239) module = importlib.import_module(module_name) 2025-10-10T01:56:39.0747171Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0747795Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:39.0748469Z (EngineCore_DP0 pid=10239) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:39.0748972Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0749476Z (EngineCore_DP0 pid=10239) File "", line 1387, in _gcd_import 2025-10-10T01:56:39.0750044Z (EngineCore_DP0 pid=10239) File "", line 1360, in _find_and_load 2025-10-10T01:56:39.0750649Z (EngineCore_DP0 pid=10239) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:39.0751246Z (EngineCore_DP0 pid=10239) File "", line 935, in _load_unlocked 2025-10-10T01:56:39.0751831Z (EngineCore_DP0 pid=10239) File "", line 999, in exec_module 2025-10-10T01:56:39.0752447Z (EngineCore_DP0 pid=10239) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:39.0753244Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:39.0753939Z (EngineCore_DP0 pid=10239) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:39.0754664Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:39.0755406Z (EngineCore_DP0 pid=10239) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:39.0756206Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:39.0756898Z (EngineCore_DP0 pid=10239) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:39.0757698Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:39.0758511Z (EngineCore_DP0 pid=10239) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:39.0758998Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0759770Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:39.0760483Z (EngineCore_DP0 pid=10239) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:39.0760934Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0761690Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:39.0762395Z (EngineCore_DP0 pid=10239) return _is_fa2_supported(device)[0] 2025-10-10T01:56:39.0762828Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0763553Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:39.0764280Z (EngineCore_DP0 pid=10239) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:39.0764811Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0765487Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:39.0766164Z (EngineCore_DP0 pid=10239) prop = get_device_properties(device) 2025-10-10T01:56:39.0766633Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:39.0767318Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:39.0767995Z (EngineCore_DP0 pid=10239) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:39.0768396Z (EngineCore_DP0 pid=10239) ^^^^^^^^^^^^ 2025-10-10T01:56:39.0768983Z (EngineCore_DP0 pid=10239) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:39.0769558Z (EngineCore_DP0 pid=10239) raise RuntimeError( 2025-10-10T01:56:39.0770254Z (EngineCore_DP0 pid=10239) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:39.4806986Z FAILED 2025-10-10T01:56:39.4936774Z models/test_initialization.py::test_can_initialize_large_subset[LlamaForCausalLMEagle3] Fork a new process to run a test 10243 2025-10-10T01:56:39.4947951Z Fork a new process to run a test 0 2025-10-10T01:56:39.5225087Z INFO 10-10 01:56:39 [utils.py:233] non-default args: {'tokenizer': 'Qwen/Qwen3-8B', 'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlamaForCausalLMEagle3', exist_overrides={}, use_original_num_layers=True), 'speculative_config': {'model': 'AngelSlim/Qwen3-8B_eagle3', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'Qwen/Qwen3-8B'} 2025-10-10T01:56:39.7196448Z INFO 10-10 01:56:39 [model.py:551] Resolved architecture: Qwen3ForCausalLM 2025-10-10T01:56:39.7196956Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:56:39.7445670Z INFO 10-10 01:56:39 [model.py:1545] Using max model len 40960 2025-10-10T01:56:39.7447393Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:56:40.0912937Z 2025-10-10T01:56:40.0913406Z config.json: 0% 0.00/763 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:40.8581920Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:40.8582627Z 2025-10-10T01:56:40.8583066Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:40.8583435Z 2025-10-10T01:56:40.8584149Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:40.8584762Z 2025-10-10T01:56:40.8585160Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:40.8585553Z 2025-10-10T01:56:40.8585912Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8586262Z 2025-10-10T01:56:40.8586883Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:40.8587482Z 2025-10-10T01:56:40.8587915Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:40.8588579Z 2025-10-10T01:56:40.8589298Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:40.8589902Z 2025-10-10T01:56:40.8590302Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:40.8590696Z 2025-10-10T01:56:40.8591056Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8591412Z 2025-10-10T01:56:40.8591942Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:40.8592427Z 2025-10-10T01:56:40.8592868Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] self._init_executor() 2025-10-10T01:56:40.8593367Z 2025-10-10T01:56:40.8594416Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:40.8595326Z 2025-10-10T01:56:40.8596295Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:40.8596677Z 2025-10-10T01:56:40.8597379Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:40.8597976Z 2025-10-10T01:56:40.8598350Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:40.8598696Z 2025-10-10T01:56:40.8598988Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8599412Z 2025-10-10T01:56:40.8599905Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:40.8600360Z 2025-10-10T01:56:40.8600643Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:40.8600926Z 2025-10-10T01:56:40.8601197Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8601461Z 2025-10-10T01:56:40.8602051Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:40.8602534Z 2025-10-10T01:56:40.8602828Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:40.8603125Z 2025-10-10T01:56:40.8603392Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8603670Z 2025-10-10T01:56:40.8604194Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:40.8604697Z 2025-10-10T01:56:40.8605005Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:40.8605314Z 2025-10-10T01:56:40.8605590Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8605864Z 2025-10-10T01:56:40.8606315Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:40.8606728Z 2025-10-10T01:56:40.8607080Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:40.8607407Z 2025-10-10T01:56:40.8607702Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8607999Z 2025-10-10T01:56:40.8608356Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:40.8608702Z 2025-10-10T01:56:40.8609056Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:40.8609405Z 2025-10-10T01:56:40.8609782Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:40.8610152Z 2025-10-10T01:56:40.8610495Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:40.8610834Z 2025-10-10T01:56:40.8611314Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:40.8611683Z 2025-10-10T01:56:40.8612113Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:40.8612516Z 2025-10-10T01:56:40.8613046Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:40.8613518Z 2025-10-10T01:56:40.8613887Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:40.8614225Z 2025-10-10T01:56:40.8614763Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:40.8615259Z 2025-10-10T01:56:40.8615634Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:40.8615998Z 2025-10-10T01:56:40.8616548Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:40.8617105Z 2025-10-10T01:56:40.8617403Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:40.8617707Z 2025-10-10T01:56:40.8618328Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:40.8618931Z 2025-10-10T01:56:40.8619314Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:40.8619670Z 2025-10-10T01:56:40.8619933Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8620200Z 2025-10-10T01:56:40.8620781Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:40.8621299Z 2025-10-10T01:56:40.8621602Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:40.8621895Z 2025-10-10T01:56:40.8622167Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8622448Z 2025-10-10T01:56:40.8623046Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:40.8623600Z 2025-10-10T01:56:40.8623882Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:40.8624177Z 2025-10-10T01:56:40.8624454Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8624721Z 2025-10-10T01:56:40.8625300Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:40.8625825Z 2025-10-10T01:56:40.8626150Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:40.8626507Z 2025-10-10T01:56:40.8626789Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8627064Z 2025-10-10T01:56:40.8627629Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:40.8628147Z 2025-10-10T01:56:40.8628434Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:40.8628728Z 2025-10-10T01:56:40.8628997Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8629278Z 2025-10-10T01:56:40.8629797Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:40.8630285Z 2025-10-10T01:56:40.8630597Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:40.8630907Z 2025-10-10T01:56:40.8631138Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:40.8631384Z 2025-10-10T01:56:40.8631908Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:40.8632357Z 2025-10-10T01:56:40.8632611Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] raise RuntimeError( 2025-10-10T01:56:40.8632871Z 2025-10-10T01:56:40.8633443Z (EngineCore_DP0 pid=10251) ERROR 10-10 01:56:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:40.8634166Z (EngineCore_DP0 pid=10251) Process EngineCore_DP0: 2025-10-10T01:56:40.8634559Z (EngineCore_DP0 pid=10251) Traceback (most recent call last): 2025-10-10T01:56:40.8635172Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:40.8635691Z (EngineCore_DP0 pid=10251) self.run() 2025-10-10T01:56:40.8636225Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:40.8636794Z (EngineCore_DP0 pid=10251) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:40.8637460Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:40.8638010Z (EngineCore_DP0 pid=10251) raise e 2025-10-10T01:56:40.8638601Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:40.8639293Z (EngineCore_DP0 pid=10251) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:40.8639750Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8640371Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:40.8641017Z (EngineCore_DP0 pid=10251) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:40.8641683Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:40.8642312Z (EngineCore_DP0 pid=10251) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:40.8642773Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8643478Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:40.8644113Z (EngineCore_DP0 pid=10251) self._init_executor() 2025-10-10T01:56:40.8644841Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:40.8645534Z (EngineCore_DP0 pid=10251) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:40.8646264Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:40.8646980Z (EngineCore_DP0 pid=10251) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:40.8647489Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8648136Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:40.8648728Z (EngineCore_DP0 pid=10251) return func(*args, **kwargs) 2025-10-10T01:56:40.8649121Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8649801Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:40.8650468Z (EngineCore_DP0 pid=10251) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:40.8650909Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8651581Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:40.8652242Z (EngineCore_DP0 pid=10251) module = importlib.import_module(module_name) 2025-10-10T01:56:40.8652682Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8653271Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:40.8653913Z (EngineCore_DP0 pid=10251) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:40.8654411Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8654905Z (EngineCore_DP0 pid=10251) File "", line 1387, in _gcd_import 2025-10-10T01:56:40.8655459Z (EngineCore_DP0 pid=10251) File "", line 1360, in _find_and_load 2025-10-10T01:56:40.8656050Z (EngineCore_DP0 pid=10251) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:40.8656631Z (EngineCore_DP0 pid=10251) File "", line 935, in _load_unlocked 2025-10-10T01:56:40.8657220Z (EngineCore_DP0 pid=10251) File "", line 999, in exec_module 2025-10-10T01:56:40.8657837Z (EngineCore_DP0 pid=10251) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:40.8658568Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:40.8659252Z (EngineCore_DP0 pid=10251) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:40.8659977Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:40.8660764Z (EngineCore_DP0 pid=10251) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:40.8661581Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:40.8662292Z (EngineCore_DP0 pid=10251) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:40.8663083Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:40.8663897Z (EngineCore_DP0 pid=10251) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:40.8664377Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8665069Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:40.8665754Z (EngineCore_DP0 pid=10251) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:40.8666193Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8666988Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:40.8667706Z (EngineCore_DP0 pid=10251) return _is_fa2_supported(device)[0] 2025-10-10T01:56:40.8668126Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8668836Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:40.8669548Z (EngineCore_DP0 pid=10251) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:40.8670003Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8670674Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:40.8671319Z (EngineCore_DP0 pid=10251) prop = get_device_properties(device) 2025-10-10T01:56:40.8671739Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:40.8672396Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:40.8673055Z (EngineCore_DP0 pid=10251) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:40.8673451Z (EngineCore_DP0 pid=10251) ^^^^^^^^^^^^ 2025-10-10T01:56:40.8674031Z (EngineCore_DP0 pid=10251) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:40.8674591Z (EngineCore_DP0 pid=10251) raise RuntimeError( 2025-10-10T01:56:40.8675278Z (EngineCore_DP0 pid=10251) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:41.2731004Z FAILED 2025-10-10T01:56:41.2859593Z models/test_initialization.py::test_can_initialize_large_subset[MiMoMTPModel] Fork a new process to run a test 10255 2025-10-10T01:56:41.2870851Z Fork a new process to run a test 0 2025-10-10T01:56:41.3144825Z INFO 10-10 01:56:41 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiMoMTPModel', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'XiaomiMiMo/MiMo-7B-RL', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'XiaomiMiMo/MiMo-7B-RL'} 2025-10-10T01:56:41.7820675Z INFO 10-10 01:56:41 [model.py:551] Resolved architecture: MiMoForCausalLM 2025-10-10T01:56:41.7821425Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:56:41.8071544Z INFO 10-10 01:56:41 [model.py:1545] Using max model len 32768 2025-10-10T01:56:41.8073251Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:56:48.6875326Z INFO 10-10 01:56:48 [model.py:551] Resolved architecture: MiMoMTPModel 2025-10-10T01:56:48.6875824Z INFO 10-10 01:56:48 [model.py:1545] Using max model len 32768 2025-10-10T01:56:48.6878268Z INFO 10-10 01:56:48 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:56:49.1215138Z (EngineCore_DP0 pid=10312) INFO 10-10 01:56:49 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:56:49.1314068Z (EngineCore_DP0 pid=10312) INFO 10-10 01:56:49 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='XiaomiMiMo/MiMo-7B-RL', speculative_config=SpeculativeConfig(method='mtp', model='XiaomiMiMo/MiMo-7B-RL', num_spec_tokens=1), tokenizer='XiaomiMiMo/MiMo-7B-RL', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=XiaomiMiMo/MiMo-7B-RL, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:56:49.2661900Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] EngineCore failed to start. 2025-10-10T01:56:49.2662562Z 2025-10-10T01:56:49.2663222Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] Traceback (most recent call last): 2025-10-10T01:56:49.2663625Z 2025-10-10T01:56:49.2664328Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:49.2664958Z 2025-10-10T01:56:49.2665345Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:49.2666015Z 2025-10-10T01:56:49.2666379Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2666731Z 2025-10-10T01:56:49.2667441Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:49.2668118Z 2025-10-10T01:56:49.2668542Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:49.2668982Z 2025-10-10T01:56:49.2669477Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:49.2669921Z 2025-10-10T01:56:49.2670238Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:49.2670546Z 2025-10-10T01:56:49.2670876Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2671286Z 2025-10-10T01:56:49.2672026Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:49.2672885Z 2025-10-10T01:56:49.2673472Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] self._init_executor() 2025-10-10T01:56:49.2673944Z 2025-10-10T01:56:49.2674908Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:49.2675791Z 2025-10-10T01:56:49.2676388Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:49.2676883Z 2025-10-10T01:56:49.2677444Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:49.2677957Z 2025-10-10T01:56:49.2678304Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:49.2678659Z 2025-10-10T01:56:49.2678948Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2679349Z 2025-10-10T01:56:49.2679841Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:49.2680313Z 2025-10-10T01:56:49.2680585Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:56:49.2680857Z 2025-10-10T01:56:49.2681115Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2681371Z 2025-10-10T01:56:49.2681895Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:49.2682382Z 2025-10-10T01:56:49.2682677Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:49.2682965Z 2025-10-10T01:56:49.2683240Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2683521Z 2025-10-10T01:56:49.2684057Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:49.2684622Z 2025-10-10T01:56:49.2684928Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:56:49.2685233Z 2025-10-10T01:56:49.2685555Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2685883Z 2025-10-10T01:56:49.2686341Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:49.2686753Z 2025-10-10T01:56:49.2687111Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:49.2687442Z 2025-10-10T01:56:49.2687735Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2688029Z 2025-10-10T01:56:49.2688432Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:56:49.2688830Z 2025-10-10T01:56:49.2689245Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:56:49.2689663Z 2025-10-10T01:56:49.2690165Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:49.2690573Z 2025-10-10T01:56:49.2690919Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:56:49.2691269Z 2025-10-10T01:56:49.2691636Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:56:49.2691989Z 2025-10-10T01:56:49.2692366Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:49.2692726Z 2025-10-10T01:56:49.2693239Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:49.2693703Z 2025-10-10T01:56:49.2694061Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:49.2694394Z 2025-10-10T01:56:49.2694919Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:49.2695399Z 2025-10-10T01:56:49.2695770Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:49.2696364Z 2025-10-10T01:56:49.2696937Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:49.2697460Z 2025-10-10T01:56:49.2697762Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:56:49.2698066Z 2025-10-10T01:56:49.2698696Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:49.2699285Z 2025-10-10T01:56:49.2699652Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:49.2700098Z 2025-10-10T01:56:49.2700365Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2700628Z 2025-10-10T01:56:49.2701260Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:49.2701842Z 2025-10-10T01:56:49.2702154Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:56:49.2702448Z 2025-10-10T01:56:49.2702721Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2703000Z 2025-10-10T01:56:49.2703618Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:49.2704174Z 2025-10-10T01:56:49.2704458Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:56:49.2704747Z 2025-10-10T01:56:49.2705008Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2705282Z 2025-10-10T01:56:49.2705932Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:49.2706468Z 2025-10-10T01:56:49.2706795Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:49.2707105Z 2025-10-10T01:56:49.2707383Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2707658Z 2025-10-10T01:56:49.2708214Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:49.2708791Z 2025-10-10T01:56:49.2709136Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:56:49.2709480Z 2025-10-10T01:56:49.2709798Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2710100Z 2025-10-10T01:56:49.2710621Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:49.2711110Z 2025-10-10T01:56:49.2711418Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:56:49.2711728Z 2025-10-10T01:56:49.2711952Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:56:49.2712194Z 2025-10-10T01:56:49.2712682Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:49.2713141Z 2025-10-10T01:56:49.2713394Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] raise RuntimeError( 2025-10-10T01:56:49.2713652Z 2025-10-10T01:56:49.2714223Z (EngineCore_DP0 pid=10312) ERROR 10-10 01:56:49 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:49.2714909Z (EngineCore_DP0 pid=10312) Process EngineCore_DP0: 2025-10-10T01:56:49.2715384Z (EngineCore_DP0 pid=10312) Traceback (most recent call last): 2025-10-10T01:56:49.2716010Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:56:49.2716561Z (EngineCore_DP0 pid=10312) self.run() 2025-10-10T01:56:49.2717124Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:56:49.2717709Z (EngineCore_DP0 pid=10312) self._target(*self._args, **self._kwargs) 2025-10-10T01:56:49.2718382Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:56:49.2719181Z (EngineCore_DP0 pid=10312) raise e 2025-10-10T01:56:49.2719860Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:56:49.2720509Z (EngineCore_DP0 pid=10312) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:56:49.2720957Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2721584Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:56:49.2722289Z (EngineCore_DP0 pid=10312) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:56:49.2722970Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:56:49.2723596Z (EngineCore_DP0 pid=10312) self.model_executor = executor_class(vllm_config) 2025-10-10T01:56:49.2724055Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2724723Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:56:49.2725311Z (EngineCore_DP0 pid=10312) self._init_executor() 2025-10-10T01:56:49.2725981Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:56:49.2726679Z (EngineCore_DP0 pid=10312) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:56:49.2727412Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:56:49.2728149Z (EngineCore_DP0 pid=10312) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:56:49.2728740Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2729513Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:56:49.2730204Z (EngineCore_DP0 pid=10312) return func(*args, **kwargs) 2025-10-10T01:56:49.2730658Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2731315Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:56:49.2731947Z (EngineCore_DP0 pid=10312) worker_class = resolve_obj_by_qualname( 2025-10-10T01:56:49.2732370Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2733035Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:56:49.2733742Z (EngineCore_DP0 pid=10312) module = importlib.import_module(module_name) 2025-10-10T01:56:49.2734189Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2734822Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:56:49.2735483Z (EngineCore_DP0 pid=10312) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:56:49.2735975Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2736468Z (EngineCore_DP0 pid=10312) File "", line 1387, in _gcd_import 2025-10-10T01:56:49.2737020Z (EngineCore_DP0 pid=10312) File "", line 1360, in _find_and_load 2025-10-10T01:56:49.2737602Z (EngineCore_DP0 pid=10312) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:56:49.2738192Z (EngineCore_DP0 pid=10312) File "", line 935, in _load_unlocked 2025-10-10T01:56:49.2738881Z (EngineCore_DP0 pid=10312) File "", line 999, in exec_module 2025-10-10T01:56:49.2739606Z (EngineCore_DP0 pid=10312) File "", line 488, in _call_with_frames_removed 2025-10-10T01:56:49.2740446Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:56:49.2741134Z (EngineCore_DP0 pid=10312) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:56:49.2741855Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:56:49.2742582Z (EngineCore_DP0 pid=10312) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:56:49.2743361Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:56:49.2744036Z (EngineCore_DP0 pid=10312) class FlashAttentionMetadataBuilder( 2025-10-10T01:56:49.2744833Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:56:49.2745662Z (EngineCore_DP0 pid=10312) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:56:49.2746147Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2746840Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:56:49.2747518Z (EngineCore_DP0 pid=10312) if not is_fa_version_supported(fa_version): 2025-10-10T01:56:49.2747959Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2748841Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:56:49.2749689Z (EngineCore_DP0 pid=10312) return _is_fa2_supported(device)[0] 2025-10-10T01:56:49.2750181Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2750891Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:56:49.2751624Z (EngineCore_DP0 pid=10312) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:56:49.2752130Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2752861Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:56:49.2753560Z (EngineCore_DP0 pid=10312) prop = get_device_properties(device) 2025-10-10T01:56:49.2753978Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:56:49.2754638Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:56:49.2755310Z (EngineCore_DP0 pid=10312) _lazy_init() # will define _get_device_properties 2025-10-10T01:56:49.2755721Z (EngineCore_DP0 pid=10312) ^^^^^^^^^^^^ 2025-10-10T01:56:49.2756315Z (EngineCore_DP0 pid=10312) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:56:49.2756885Z (EngineCore_DP0 pid=10312) raise RuntimeError( 2025-10-10T01:56:49.2757578Z (EngineCore_DP0 pid=10312) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:56:49.6781832Z FAILED 2025-10-10T01:56:49.6911211Z models/test_initialization.py::test_can_initialize_large_subset[GPTJForCausalLM] Fork a new process to run a test 10316 2025-10-10T01:56:49.6921884Z Fork a new process to run a test 0 2025-10-10T01:56:49.7210431Z INFO 10-10 01:56:49 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GPTJForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Milos/slovak-gpt-j-405M'} 2025-10-10T01:56:49.8511679Z 2025-10-10T01:56:49.8513063Z config.json: 0% 0.00/836 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:57:00.7338462Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] EngineCore failed to start. 2025-10-10T01:57:00.7338880Z 2025-10-10T01:57:00.7339260Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] Traceback (most recent call last): 2025-10-10T01:57:00.7339854Z 2025-10-10T01:57:00.7340539Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:00.7341167Z 2025-10-10T01:57:00.7341554Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:00.7341930Z 2025-10-10T01:57:00.7342285Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7342635Z 2025-10-10T01:57:00.7343245Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:00.7343933Z 2025-10-10T01:57:00.7344417Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:00.7344819Z 2025-10-10T01:57:00.7345425Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:00.7345971Z 2025-10-10T01:57:00.7346362Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:00.7346747Z 2025-10-10T01:57:00.7347231Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7347843Z 2025-10-10T01:57:00.7349053Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:00.7349992Z 2025-10-10T01:57:00.7350509Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] self._init_executor() 2025-10-10T01:57:00.7351015Z 2025-10-10T01:57:00.7351614Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:00.7352145Z 2025-10-10T01:57:00.7352479Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:00.7352934Z 2025-10-10T01:57:00.7353496Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:00.7354001Z 2025-10-10T01:57:00.7354441Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:00.7354850Z 2025-10-10T01:57:00.7355152Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7355473Z 2025-10-10T01:57:00.7355975Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:00.7356439Z 2025-10-10T01:57:00.7356707Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:57:00.7356992Z 2025-10-10T01:57:00.7357247Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7357506Z 2025-10-10T01:57:00.7358030Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:00.7358507Z 2025-10-10T01:57:00.7358849Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:00.7359274Z 2025-10-10T01:57:00.7359557Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7359828Z 2025-10-10T01:57:00.7360360Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:00.7360855Z 2025-10-10T01:57:00.7361157Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:57:00.7361462Z 2025-10-10T01:57:00.7361736Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7362022Z 2025-10-10T01:57:00.7362464Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:00.7362884Z 2025-10-10T01:57:00.7363226Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:00.7363555Z 2025-10-10T01:57:00.7363846Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7364128Z 2025-10-10T01:57:00.7364473Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:57:00.7364802Z 2025-10-10T01:57:00.7365158Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:57:00.7365499Z 2025-10-10T01:57:00.7365873Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:00.7366246Z 2025-10-10T01:57:00.7366588Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:57:00.7366933Z 2025-10-10T01:57:00.7367304Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:57:00.7367711Z 2025-10-10T01:57:00.7368097Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:00.7368465Z 2025-10-10T01:57:00.7369033Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:00.7369543Z 2025-10-10T01:57:00.7369903Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:00.7370236Z 2025-10-10T01:57:00.7370763Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:00.7371242Z 2025-10-10T01:57:00.7371629Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:00.7372001Z 2025-10-10T01:57:00.7372552Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:00.7373083Z 2025-10-10T01:57:00.7373380Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:57:00.7373723Z 2025-10-10T01:57:00.7374362Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:00.7374954Z 2025-10-10T01:57:00.7375319Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:00.7375682Z 2025-10-10T01:57:00.7375938Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7376197Z 2025-10-10T01:57:00.7376771Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:00.7377290Z 2025-10-10T01:57:00.7377596Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:57:00.7377891Z 2025-10-10T01:57:00.7378167Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7378437Z 2025-10-10T01:57:00.7379056Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:00.7379631Z 2025-10-10T01:57:00.7379917Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:57:00.7380210Z 2025-10-10T01:57:00.7380474Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7380765Z 2025-10-10T01:57:00.7381372Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:00.7381927Z 2025-10-10T01:57:00.7382252Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:00.7382575Z 2025-10-10T01:57:00.7382850Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7383175Z 2025-10-10T01:57:00.7383713Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:00.7384202Z 2025-10-10T01:57:00.7384532Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:57:00.7384896Z 2025-10-10T01:57:00.7385181Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7385462Z 2025-10-10T01:57:00.7385999Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:00.7386496Z 2025-10-10T01:57:00.7386819Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:57:00.7387144Z 2025-10-10T01:57:00.7387376Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:57:00.7387630Z 2025-10-10T01:57:00.7388125Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:00.7388586Z 2025-10-10T01:57:00.7388879Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] raise RuntimeError( 2025-10-10T01:57:00.7389150Z 2025-10-10T01:57:00.7389730Z (EngineCore_DP0 pid=10374) ERROR 10-10 01:57:00 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:00.7390437Z (EngineCore_DP0 pid=10374) Process EngineCore_DP0: 2025-10-10T01:57:00.7390834Z (EngineCore_DP0 pid=10374) Traceback (most recent call last): 2025-10-10T01:57:00.7391478Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:57:00.7392003Z (EngineCore_DP0 pid=10374) self.run() 2025-10-10T01:57:00.7392555Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:57:00.7393135Z (EngineCore_DP0 pid=10374) self._target(*self._args, **self._kwargs) 2025-10-10T01:57:00.7393816Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:57:00.7394370Z (EngineCore_DP0 pid=10374) raise e 2025-10-10T01:57:00.7394968Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:00.7395628Z (EngineCore_DP0 pid=10374) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:00.7396276Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7396922Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:00.7397574Z (EngineCore_DP0 pid=10374) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:00.7398237Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:00.7398870Z (EngineCore_DP0 pid=10374) self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:00.7399406Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7400066Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:00.7400756Z (EngineCore_DP0 pid=10374) self._init_executor() 2025-10-10T01:57:00.7401489Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:00.7402250Z (EngineCore_DP0 pid=10374) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:00.7403003Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:00.7403760Z (EngineCore_DP0 pid=10374) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:00.7404441Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7405091Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:00.7405688Z (EngineCore_DP0 pid=10374) return func(*args, **kwargs) 2025-10-10T01:57:00.7406082Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7406728Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:00.7407468Z (EngineCore_DP0 pid=10374) worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:00.7407915Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7408592Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:00.7409261Z (EngineCore_DP0 pid=10374) module = importlib.import_module(module_name) 2025-10-10T01:57:00.7409720Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7410314Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:00.7410941Z (EngineCore_DP0 pid=10374) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:00.7411447Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7411949Z (EngineCore_DP0 pid=10374) File "", line 1387, in _gcd_import 2025-10-10T01:57:00.7412512Z (EngineCore_DP0 pid=10374) File "", line 1360, in _find_and_load 2025-10-10T01:57:00.7413123Z (EngineCore_DP0 pid=10374) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:00.7413712Z (EngineCore_DP0 pid=10374) File "", line 935, in _load_unlocked 2025-10-10T01:57:00.7414284Z (EngineCore_DP0 pid=10374) File "", line 999, in exec_module 2025-10-10T01:57:00.7414899Z (EngineCore_DP0 pid=10374) File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:00.7415648Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:00.7416335Z (EngineCore_DP0 pid=10374) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:00.7417062Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:00.7417784Z (EngineCore_DP0 pid=10374) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:00.7418625Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:00.7419355Z (EngineCore_DP0 pid=10374) class FlashAttentionMetadataBuilder( 2025-10-10T01:57:00.7420200Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:00.7421020Z (EngineCore_DP0 pid=10374) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:00.7421511Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7422202Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:00.7422901Z (EngineCore_DP0 pid=10374) if not is_fa_version_supported(fa_version): 2025-10-10T01:57:00.7423350Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7424103Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:00.7424858Z (EngineCore_DP0 pid=10374) return _is_fa2_supported(device)[0] 2025-10-10T01:57:00.7425276Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7425996Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:00.7426714Z (EngineCore_DP0 pid=10374) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:00.7427172Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7427849Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:00.7428485Z (EngineCore_DP0 pid=10374) prop = get_device_properties(device) 2025-10-10T01:57:00.7428921Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:00.7429583Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:00.7430263Z (EngineCore_DP0 pid=10374) _lazy_init() # will define _get_device_properties 2025-10-10T01:57:00.7430671Z (EngineCore_DP0 pid=10374) ^^^^^^^^^^^^ 2025-10-10T01:57:00.7431250Z (EngineCore_DP0 pid=10374) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:00.7431820Z (EngineCore_DP0 pid=10374) raise RuntimeError( 2025-10-10T01:57:00.7432511Z (EngineCore_DP0 pid=10374) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:01.1406858Z FAILED 2025-10-10T01:57:01.1537038Z models/test_initialization.py::test_can_initialize_large_subset[ModernBertForSequenceClassification] Fork a new process to run a test 10378 2025-10-10T01:57:01.1548555Z Fork a new process to run a test 0 2025-10-10T01:57:01.1824602Z INFO 10-10 01:57:01 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ModernBertForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Alibaba-NLP/gte-reranker-modernbert-base'} 2025-10-10T01:57:02.0141428Z 2025-10-10T01:57:02.0143259Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:57:02.0143562Z config.json: 1.33kB [00:00, 6.72MB/s] 2025-10-10T01:57:09.7093999Z INFO 10-10 01:57:09 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:57:09.7094987Z INFO 10-10 01:57:09 [model.py:551] Resolved architecture: ModernBertForSequenceClassification 2025-10-10T01:57:09.7095617Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:57:09.7341045Z INFO 10-10 01:57:09 [model.py:1765] Downcasting torch.float32 to torch.float16. 2025-10-10T01:57:11.3894959Z 2025-10-10T01:57:11.3897141Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:57:11.3897523Z tokenizer_config.json: 21.0kB [00:00, 111MB/s] 2025-10-10T01:57:11.3906228Z INFO 10-10 01:57:11 [model.py:1545] Using max model len 8192 2025-10-10T01:57:11.4775122Z INFO 10-10 01:57:11 [arg_utils.py:1580] (Disabling) chunked prefill by default 2025-10-10T01:57:11.4775648Z INFO 10-10 01:57:11 [arg_utils.py:1583] (Disabling) prefix caching by default 2025-10-10T01:57:11.6213699Z INFO 10-10 01:57:11 [vllm.py:404] Only "last" pooling supports chunked prefill and prefix caching; disabling both. 2025-10-10T01:57:12.3182375Z 2025-10-10T01:57:12.3873219Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:57:12.3873629Z tokenizer.json: 3.58MB [00:00, 51.9MB/s] 2025-10-10T01:57:13.3206069Z 2025-10-10T01:57:13.3206462Z special_tokens_map.json: 0% 0.00/694 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:57:14.3922091Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] EngineCore failed to start. 2025-10-10T01:57:14.3922672Z 2025-10-10T01:57:14.3923489Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] Traceback (most recent call last): 2025-10-10T01:57:14.3924144Z 2025-10-10T01:57:14.3925136Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:14.3926014Z 2025-10-10T01:57:14.3926593Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:14.3927178Z 2025-10-10T01:57:14.3927710Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3928281Z 2025-10-10T01:57:14.3929259Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:14.3930161Z 2025-10-10T01:57:14.3930869Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:14.3931520Z 2025-10-10T01:57:14.3932661Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:14.3933573Z 2025-10-10T01:57:14.3934226Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:14.3934830Z 2025-10-10T01:57:14.3935427Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3935981Z 2025-10-10T01:57:14.3936957Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:14.3937875Z 2025-10-10T01:57:14.3938366Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] self._init_executor() 2025-10-10T01:57:14.3938857Z 2025-10-10T01:57:14.3939766Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:14.3940649Z 2025-10-10T01:57:14.3941273Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:14.3941784Z 2025-10-10T01:57:14.3942499Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:14.3943037Z 2025-10-10T01:57:14.3943418Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:14.3943759Z 2025-10-10T01:57:14.3944063Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3944348Z 2025-10-10T01:57:14.3944850Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:14.3945314Z 2025-10-10T01:57:14.3945587Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:57:14.3945871Z 2025-10-10T01:57:14.3946291Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3946563Z 2025-10-10T01:57:14.3947155Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:14.3947690Z 2025-10-10T01:57:14.3947989Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:14.3948288Z 2025-10-10T01:57:14.3948558Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3948830Z 2025-10-10T01:57:14.3949393Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:14.3949896Z 2025-10-10T01:57:14.3950211Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:57:14.3950513Z 2025-10-10T01:57:14.3950796Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3951073Z 2025-10-10T01:57:14.3951580Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:14.3952006Z 2025-10-10T01:57:14.3952353Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:14.3952690Z 2025-10-10T01:57:14.3952979Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3953266Z 2025-10-10T01:57:14.3953612Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:57:14.3953954Z 2025-10-10T01:57:14.3954306Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:57:14.3954649Z 2025-10-10T01:57:14.3955045Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:14.3955405Z 2025-10-10T01:57:14.3955753Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:57:14.3956086Z 2025-10-10T01:57:14.3956456Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:57:14.3956806Z 2025-10-10T01:57:14.3957183Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:14.3957551Z 2025-10-10T01:57:14.3958049Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:14.3958517Z 2025-10-10T01:57:14.3958873Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:14.3959347Z 2025-10-10T01:57:14.3959870Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:14.3960353Z 2025-10-10T01:57:14.3960732Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:14.3961152Z 2025-10-10T01:57:14.3961710Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:14.3962229Z 2025-10-10T01:57:14.3962581Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:57:14.3962922Z 2025-10-10T01:57:14.3963553Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:14.3964127Z 2025-10-10T01:57:14.3964499Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:14.3964851Z 2025-10-10T01:57:14.3965104Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3965373Z 2025-10-10T01:57:14.3965934Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:14.3966460Z 2025-10-10T01:57:14.3966796Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:57:14.3967106Z 2025-10-10T01:57:14.3967392Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3967675Z 2025-10-10T01:57:14.3968277Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:14.3968843Z 2025-10-10T01:57:14.3969132Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:57:14.3969416Z 2025-10-10T01:57:14.3969685Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3969954Z 2025-10-10T01:57:14.3970540Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:14.3971065Z 2025-10-10T01:57:14.3971383Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:14.3971699Z 2025-10-10T01:57:14.3971972Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3972250Z 2025-10-10T01:57:14.3972773Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:14.3973262Z 2025-10-10T01:57:14.3973547Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:57:14.3973841Z 2025-10-10T01:57:14.3974114Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3974387Z 2025-10-10T01:57:14.3974917Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:14.3975400Z 2025-10-10T01:57:14.3975727Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:57:14.3976036Z 2025-10-10T01:57:14.3976340Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:57:14.3976592Z 2025-10-10T01:57:14.3977138Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:14.3977664Z 2025-10-10T01:57:14.3977917Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] raise RuntimeError( 2025-10-10T01:57:14.3978189Z 2025-10-10T01:57:14.3978760Z (EngineCore_DP0 pid=10437) ERROR 10-10 01:57:14 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:14.3979473Z (EngineCore_DP0 pid=10437) Process EngineCore_DP0: 2025-10-10T01:57:14.3979906Z (EngineCore_DP0 pid=10437) Traceback (most recent call last): 2025-10-10T01:57:14.3980522Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:57:14.3981046Z (EngineCore_DP0 pid=10437) self.run() 2025-10-10T01:57:14.3981579Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:57:14.3982162Z (EngineCore_DP0 pid=10437) self._target(*self._args, **self._kwargs) 2025-10-10T01:57:14.3982876Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:57:14.3983437Z (EngineCore_DP0 pid=10437) raise e 2025-10-10T01:57:14.3984057Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:14.3984717Z (EngineCore_DP0 pid=10437) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:14.3985171Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3985802Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:14.3986468Z (EngineCore_DP0 pid=10437) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:14.3987141Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:14.3987772Z (EngineCore_DP0 pid=10437) self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:14.3988226Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3988885Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:14.3989488Z (EngineCore_DP0 pid=10437) self._init_executor() 2025-10-10T01:57:14.3990153Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:14.3990845Z (EngineCore_DP0 pid=10437) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:14.3991572Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:14.3992285Z (EngineCore_DP0 pid=10437) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:14.3992799Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3993446Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:14.3994090Z (EngineCore_DP0 pid=10437) return func(*args, **kwargs) 2025-10-10T01:57:14.3994485Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3995175Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:14.3995860Z (EngineCore_DP0 pid=10437) worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:14.3996709Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3997622Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:14.3998286Z (EngineCore_DP0 pid=10437) module = importlib.import_module(module_name) 2025-10-10T01:57:14.3998742Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.3999400Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:14.4000039Z (EngineCore_DP0 pid=10437) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:14.4000536Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.4001126Z (EngineCore_DP0 pid=10437) File "", line 1387, in _gcd_import 2025-10-10T01:57:14.4001707Z (EngineCore_DP0 pid=10437) File "", line 1360, in _find_and_load 2025-10-10T01:57:14.4002315Z (EngineCore_DP0 pid=10437) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:14.4002911Z (EngineCore_DP0 pid=10437) File "", line 935, in _load_unlocked 2025-10-10T01:57:14.4003491Z (EngineCore_DP0 pid=10437) File "", line 999, in exec_module 2025-10-10T01:57:14.4004102Z (EngineCore_DP0 pid=10437) File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:14.4004842Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:14.4005533Z (EngineCore_DP0 pid=10437) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:14.4006272Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:14.4007004Z (EngineCore_DP0 pid=10437) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:14.4007802Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:14.4008469Z (EngineCore_DP0 pid=10437) class FlashAttentionMetadataBuilder( 2025-10-10T01:57:14.4009272Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:14.4010118Z (EngineCore_DP0 pid=10437) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:14.4010605Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.4011305Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:14.4012008Z (EngineCore_DP0 pid=10437) if not is_fa_version_supported(fa_version): 2025-10-10T01:57:14.4012544Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.4013356Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:14.4014141Z (EngineCore_DP0 pid=10437) return _is_fa2_supported(device)[0] 2025-10-10T01:57:14.4014563Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.4015285Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:14.4016002Z (EngineCore_DP0 pid=10437) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:14.4016462Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.4017145Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:14.4017795Z (EngineCore_DP0 pid=10437) prop = get_device_properties(device) 2025-10-10T01:57:14.4018223Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:14.4018925Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:14.4019598Z (EngineCore_DP0 pid=10437) _lazy_init() # will define _get_device_properties 2025-10-10T01:57:14.4019998Z (EngineCore_DP0 pid=10437) ^^^^^^^^^^^^ 2025-10-10T01:57:14.4020580Z (EngineCore_DP0 pid=10437) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:14.4021146Z (EngineCore_DP0 pid=10437) raise RuntimeError( 2025-10-10T01:57:14.4021832Z (EngineCore_DP0 pid=10437) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:14.8054608Z FAILED 2025-10-10T01:57:14.8183222Z models/test_initialization.py::test_can_initialize_large_subset[GLM4VForCausalLM] Fork a new process to run a test 10441 2025-10-10T01:57:14.8194125Z Fork a new process to run a test 0 2025-10-10T01:57:14.8475185Z INFO 10-10 01:57:14 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GLM4VForCausalLM', exist_overrides={'architectures': ['GLM4VForCausalLM']}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'zai-org/glm-4v-9b'} 2025-10-10T01:57:16.4849014Z 2025-10-10T01:57:16.4851136Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:57:16.4851549Z config.json: 1.77kB [00:00, 8.40MB/s] 2025-10-10T01:57:25.0989691Z INFO 10-10 01:57:25 [model.py:551] Resolved architecture: GLM4VForCausalLM 2025-10-10T01:57:25.0990180Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:57:25.1241798Z INFO 10-10 01:57:25 [model.py:1545] Using max model len 8192 2025-10-10T01:57:25.1244332Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T01:57:25.4504456Z INFO 10-10 01:57:25 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:57:25.8246465Z 2025-10-10T01:57:25.8247792Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:57:25.8248168Z tokenizer_config.json: 3.22kB [00:00, 25.6MB/s] 2025-10-10T01:57:25.8833209Z 2025-10-10T01:57:25.8835495Z tokenization_chatglm.py: 0.00B [00:00, ?B/s] 2025-10-10T01:57:25.8835873Z tokenization_chatglm.py: 17.7kB [00:00, 46.9MB/s] 2025-10-10T01:57:25.8976937Z A new version of the following files was downloaded from https://huggingface.co/zai-org/glm-4v-9b: 2025-10-10T01:57:25.8977573Z - tokenization_chatglm.py 2025-10-10T01:57:25.8978337Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:57:26.3034020Z 2025-10-10T01:57:26.5046627Z tokenizer.model: 0% 0.00/2.62M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:57:30.0060351Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] EngineCore failed to start. 2025-10-10T01:57:30.0060840Z 2025-10-10T01:57:30.0061235Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] Traceback (most recent call last): 2025-10-10T01:57:30.0061610Z 2025-10-10T01:57:30.0062273Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:30.0063188Z 2025-10-10T01:57:30.0063586Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:30.0063969Z 2025-10-10T01:57:30.0064451Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0064803Z 2025-10-10T01:57:30.0065540Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:30.0066134Z 2025-10-10T01:57:30.0066642Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:30.0066983Z 2025-10-10T01:57:30.0067473Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:30.0067925Z 2025-10-10T01:57:30.0068237Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:30.0068548Z 2025-10-10T01:57:30.0068828Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0069113Z 2025-10-10T01:57:30.0069718Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:30.0070296Z 2025-10-10T01:57:30.0070762Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] self._init_executor() 2025-10-10T01:57:30.0071048Z 2025-10-10T01:57:30.0072023Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:30.0072630Z 2025-10-10T01:57:30.0073081Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:30.0073435Z 2025-10-10T01:57:30.0073987Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:30.0074489Z 2025-10-10T01:57:30.0074839Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:30.0075175Z 2025-10-10T01:57:30.0075471Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0075758Z 2025-10-10T01:57:30.0076249Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:30.0076700Z 2025-10-10T01:57:30.0076984Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:57:30.0077259Z 2025-10-10T01:57:30.0077523Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0077807Z 2025-10-10T01:57:30.0078358Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:30.0078862Z 2025-10-10T01:57:30.0079283Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:30.0079585Z 2025-10-10T01:57:30.0079865Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0080224Z 2025-10-10T01:57:30.0080773Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:30.0081264Z 2025-10-10T01:57:30.0081677Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:57:30.0082022Z 2025-10-10T01:57:30.0082308Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0082615Z 2025-10-10T01:57:30.0083062Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:30.0083479Z 2025-10-10T01:57:30.0083822Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:30.0084159Z 2025-10-10T01:57:30.0084445Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0084729Z 2025-10-10T01:57:30.0085080Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:57:30.0091938Z 2025-10-10T01:57:30.0092502Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:57:30.0092892Z 2025-10-10T01:57:30.0093308Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:30.0093693Z 2025-10-10T01:57:30.0094048Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:57:30.0094394Z 2025-10-10T01:57:30.0094769Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:57:30.0095125Z 2025-10-10T01:57:30.0095511Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:30.0095891Z 2025-10-10T01:57:30.0096764Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:30.0097246Z 2025-10-10T01:57:30.0097636Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:30.0097987Z 2025-10-10T01:57:30.0098527Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:30.0099037Z 2025-10-10T01:57:30.0099416Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:30.0099786Z 2025-10-10T01:57:30.0100358Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:30.0100884Z 2025-10-10T01:57:30.0101183Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:57:30.0101482Z 2025-10-10T01:57:30.0102123Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:30.0102816Z 2025-10-10T01:57:30.0103198Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:30.0103553Z 2025-10-10T01:57:30.0103897Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0104221Z 2025-10-10T01:57:30.0104798Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:30.0105316Z 2025-10-10T01:57:30.0105615Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:57:30.0105930Z 2025-10-10T01:57:30.0106205Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0106489Z 2025-10-10T01:57:30.0107087Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:30.0107656Z 2025-10-10T01:57:30.0107945Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:57:30.0108239Z 2025-10-10T01:57:30.0108571Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0108848Z 2025-10-10T01:57:30.0109442Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:30.0109969Z 2025-10-10T01:57:30.0110294Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:30.0110610Z 2025-10-10T01:57:30.0110891Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0111167Z 2025-10-10T01:57:30.0111693Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:30.0112184Z 2025-10-10T01:57:30.0112470Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:57:30.0112766Z 2025-10-10T01:57:30.0113045Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0113329Z 2025-10-10T01:57:30.0113851Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:30.0114340Z 2025-10-10T01:57:30.0114654Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:57:30.0114963Z 2025-10-10T01:57:30.0115200Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:57:30.0115448Z 2025-10-10T01:57:30.0115936Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:30.0116385Z 2025-10-10T01:57:30.0116643Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] raise RuntimeError( 2025-10-10T01:57:30.0116904Z 2025-10-10T01:57:30.0117478Z (EngineCore_DP0 pid=10519) ERROR 10-10 01:57:30 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:30.0118227Z (EngineCore_DP0 pid=10519) Process EngineCore_DP0: 2025-10-10T01:57:30.0118644Z (EngineCore_DP0 pid=10519) Traceback (most recent call last): 2025-10-10T01:57:30.0119404Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:57:30.0119970Z (EngineCore_DP0 pid=10519) self.run() 2025-10-10T01:57:30.0120497Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:57:30.0121070Z (EngineCore_DP0 pid=10519) self._target(*self._args, **self._kwargs) 2025-10-10T01:57:30.0121742Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:57:30.0122298Z (EngineCore_DP0 pid=10519) raise e 2025-10-10T01:57:30.0122908Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:30.0123552Z (EngineCore_DP0 pid=10519) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:30.0124009Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0124687Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:30.0125351Z (EngineCore_DP0 pid=10519) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:30.0126021Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:30.0126651Z (EngineCore_DP0 pid=10519) self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:30.0127109Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0127786Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:30.0128387Z (EngineCore_DP0 pid=10519) self._init_executor() 2025-10-10T01:57:30.0129092Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:30.0129809Z (EngineCore_DP0 pid=10519) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:30.0130538Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:30.0131261Z (EngineCore_DP0 pid=10519) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:30.0131766Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0132424Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:30.0133030Z (EngineCore_DP0 pid=10519) return func(*args, **kwargs) 2025-10-10T01:57:30.0133422Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0134065Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:30.0134707Z (EngineCore_DP0 pid=10519) worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:30.0135143Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0135821Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:30.0136526Z (EngineCore_DP0 pid=10519) module = importlib.import_module(module_name) 2025-10-10T01:57:30.0137020Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0137653Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:30.0138286Z (EngineCore_DP0 pid=10519) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:30.0138781Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0139272Z (EngineCore_DP0 pid=10519) File "", line 1387, in _gcd_import 2025-10-10T01:57:30.0139832Z (EngineCore_DP0 pid=10519) File "", line 1360, in _find_and_load 2025-10-10T01:57:30.0140423Z (EngineCore_DP0 pid=10519) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:30.0141007Z (EngineCore_DP0 pid=10519) File "", line 935, in _load_unlocked 2025-10-10T01:57:30.0141586Z (EngineCore_DP0 pid=10519) File "", line 999, in exec_module 2025-10-10T01:57:30.0142243Z (EngineCore_DP0 pid=10519) File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:30.0142981Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:30.0143662Z (EngineCore_DP0 pid=10519) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:30.0144386Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:30.0145114Z (EngineCore_DP0 pid=10519) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:30.0145909Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:30.0146584Z (EngineCore_DP0 pid=10519) class FlashAttentionMetadataBuilder( 2025-10-10T01:57:30.0147378Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:30.0148200Z (EngineCore_DP0 pid=10519) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:30.0148689Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0149430Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:30.0150132Z (EngineCore_DP0 pid=10519) if not is_fa_version_supported(fa_version): 2025-10-10T01:57:30.0150577Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0151336Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:30.0152048Z (EngineCore_DP0 pid=10519) return _is_fa2_supported(device)[0] 2025-10-10T01:57:30.0152474Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0153193Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:30.0153954Z (EngineCore_DP0 pid=10519) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:30.0154421Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0155161Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:30.0155846Z (EngineCore_DP0 pid=10519) prop = get_device_properties(device) 2025-10-10T01:57:30.0156275Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:30.0156944Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:30.0157611Z (EngineCore_DP0 pid=10519) _lazy_init() # will define _get_device_properties 2025-10-10T01:57:30.0158027Z (EngineCore_DP0 pid=10519) ^^^^^^^^^^^^ 2025-10-10T01:57:30.0158614Z (EngineCore_DP0 pid=10519) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:30.0159278Z (EngineCore_DP0 pid=10519) raise RuntimeError( 2025-10-10T01:57:30.0160026Z (EngineCore_DP0 pid=10519) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:30.4277828Z FAILED 2025-10-10T01:57:30.4408573Z models/test_initialization.py::test_can_initialize_large_subset[DeepseekVLV2ForCausalLM] Fork a new process to run a test 10523 2025-10-10T01:57:30.4419649Z Fork a new process to run a test 0 2025-10-10T01:57:30.4423635Z `transformers==4.56.2` installed, but `transformers<=4.48` is required to run this model. Reason: HF model is not compatible. 2025-10-10T01:57:30.7443046Z PASSED 2025-10-10T01:57:30.7573461Z models/test_initialization.py::test_can_initialize_large_subset[ChameleonForConditionalGeneration] Fork a new process to run a test 10524 2025-10-10T01:57:30.7584718Z Fork a new process to run a test 0 2025-10-10T01:57:30.7866597Z INFO 10-10 01:57:30 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='ChameleonForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'facebook/chameleon-7b'} 2025-10-10T01:57:32.8194741Z 2025-10-10T01:57:32.8497318Z config.json: 0% 0.00/1.71M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:57:49.8346097Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] EngineCore failed to start. 2025-10-10T01:57:49.8346709Z 2025-10-10T01:57:49.8347415Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] Traceback (most recent call last): 2025-10-10T01:57:49.8348062Z 2025-10-10T01:57:49.8349242Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:49.8350010Z 2025-10-10T01:57:49.8350440Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:49.8350850Z 2025-10-10T01:57:49.8351210Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8351568Z 2025-10-10T01:57:49.8352192Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:49.8352756Z 2025-10-10T01:57:49.8353176Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:49.8353802Z 2025-10-10T01:57:49.8354371Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:49.8354810Z 2025-10-10T01:57:49.8355224Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:49.8355621Z 2025-10-10T01:57:49.8355909Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8356192Z 2025-10-10T01:57:49.8356696Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:49.8357181Z 2025-10-10T01:57:49.8357435Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] self._init_executor() 2025-10-10T01:57:49.8357703Z 2025-10-10T01:57:49.8358243Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:49.8358737Z 2025-10-10T01:57:49.8359200Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:49.8359525Z 2025-10-10T01:57:49.8360154Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:49.8360686Z 2025-10-10T01:57:49.8361043Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:49.8361374Z 2025-10-10T01:57:49.8361664Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8361951Z 2025-10-10T01:57:49.8362442Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:49.8362907Z 2025-10-10T01:57:49.8363180Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:57:49.8363456Z 2025-10-10T01:57:49.8363710Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8364059Z 2025-10-10T01:57:49.8364926Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:49.8365746Z 2025-10-10T01:57:49.8366243Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:49.8366670Z 2025-10-10T01:57:49.8366960Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8367289Z 2025-10-10T01:57:49.8367932Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:49.8368598Z 2025-10-10T01:57:49.8369152Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:57:49.8369549Z 2025-10-10T01:57:49.8369850Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8370130Z 2025-10-10T01:57:49.8370585Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:49.8371096Z 2025-10-10T01:57:49.8371441Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:49.8371784Z 2025-10-10T01:57:49.8372112Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8372431Z 2025-10-10T01:57:49.8372779Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:57:49.8373113Z 2025-10-10T01:57:49.8373499Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:57:49.8373901Z 2025-10-10T01:57:49.8374350Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:49.8374797Z 2025-10-10T01:57:49.8375213Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:57:49.8375605Z 2025-10-10T01:57:49.8375996Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:57:49.8376349Z 2025-10-10T01:57:49.8376759Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:49.8377127Z 2025-10-10T01:57:49.8377626Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:49.8378091Z 2025-10-10T01:57:49.8378441Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:49.8378782Z 2025-10-10T01:57:49.8379300Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:49.8379779Z 2025-10-10T01:57:49.8380150Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:49.8380510Z 2025-10-10T01:57:49.8381053Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:49.8381557Z 2025-10-10T01:57:49.8381870Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:57:49.8382166Z 2025-10-10T01:57:49.8382791Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:49.8383390Z 2025-10-10T01:57:49.8383834Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:49.8384253Z 2025-10-10T01:57:49.8384559Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8384875Z 2025-10-10T01:57:49.8385468Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:49.8385985Z 2025-10-10T01:57:49.8386281Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:57:49.8386627Z 2025-10-10T01:57:49.8386905Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8387188Z 2025-10-10T01:57:49.8387851Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:49.8388439Z 2025-10-10T01:57:49.8388732Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:57:49.8389017Z 2025-10-10T01:57:49.8389286Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8389553Z 2025-10-10T01:57:49.8390142Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:49.8390668Z 2025-10-10T01:57:49.8390998Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:49.8391309Z 2025-10-10T01:57:49.8391586Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8391869Z 2025-10-10T01:57:49.8392432Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:49.8392934Z 2025-10-10T01:57:49.8393221Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:57:49.8393516Z 2025-10-10T01:57:49.8393783Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8394067Z 2025-10-10T01:57:49.8394594Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:49.8395075Z 2025-10-10T01:57:49.8395394Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:57:49.8395699Z 2025-10-10T01:57:49.8395931Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:57:49.8396476Z 2025-10-10T01:57:49.8396976Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:49.8397434Z 2025-10-10T01:57:49.8397684Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] raise RuntimeError( 2025-10-10T01:57:49.8397954Z 2025-10-10T01:57:49.8398520Z (EngineCore_DP0 pid=10582) ERROR 10-10 01:57:49 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:49.8399316Z (EngineCore_DP0 pid=10582) Process EngineCore_DP0: 2025-10-10T01:57:49.8399732Z (EngineCore_DP0 pid=10582) Traceback (most recent call last): 2025-10-10T01:57:49.8400363Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:57:49.8400883Z (EngineCore_DP0 pid=10582) self.run() 2025-10-10T01:57:49.8401431Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:57:49.8402012Z (EngineCore_DP0 pid=10582) self._target(*self._args, **self._kwargs) 2025-10-10T01:57:49.8402795Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:57:49.8403353Z (EngineCore_DP0 pid=10582) raise e 2025-10-10T01:57:49.8404032Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:49.8404763Z (EngineCore_DP0 pid=10582) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:49.8405211Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8405839Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:49.8406482Z (EngineCore_DP0 pid=10582) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:49.8407148Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:49.8407761Z (EngineCore_DP0 pid=10582) self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:49.8408217Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8408934Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:49.8409529Z (EngineCore_DP0 pid=10582) self._init_executor() 2025-10-10T01:57:49.8410210Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:49.8410903Z (EngineCore_DP0 pid=10582) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:49.8411626Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:49.8412337Z (EngineCore_DP0 pid=10582) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:49.8412837Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8413483Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:49.8414073Z (EngineCore_DP0 pid=10582) return func(*args, **kwargs) 2025-10-10T01:57:49.8414457Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8415093Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:49.8415746Z (EngineCore_DP0 pid=10582) worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:49.8416177Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8416855Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:49.8417512Z (EngineCore_DP0 pid=10582) module = importlib.import_module(module_name) 2025-10-10T01:57:49.8417962Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8418546Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:49.8419185Z (EngineCore_DP0 pid=10582) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:49.8419681Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8420229Z (EngineCore_DP0 pid=10582) File "", line 1387, in _gcd_import 2025-10-10T01:57:49.8420786Z (EngineCore_DP0 pid=10582) File "", line 1360, in _find_and_load 2025-10-10T01:57:49.8421435Z (EngineCore_DP0 pid=10582) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:49.8422074Z (EngineCore_DP0 pid=10582) File "", line 935, in _load_unlocked 2025-10-10T01:57:49.8422653Z (EngineCore_DP0 pid=10582) File "", line 999, in exec_module 2025-10-10T01:57:49.8423255Z (EngineCore_DP0 pid=10582) File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:49.8423998Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:49.8424686Z (EngineCore_DP0 pid=10582) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:49.8425417Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:49.8426157Z (EngineCore_DP0 pid=10582) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:49.8426991Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:49.8427665Z (EngineCore_DP0 pid=10582) class FlashAttentionMetadataBuilder( 2025-10-10T01:57:49.8428458Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:49.8429274Z (EngineCore_DP0 pid=10582) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:49.8429752Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8430450Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:49.8431146Z (EngineCore_DP0 pid=10582) if not is_fa_version_supported(fa_version): 2025-10-10T01:57:49.8431590Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8432335Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:49.8433035Z (EngineCore_DP0 pid=10582) return _is_fa2_supported(device)[0] 2025-10-10T01:57:49.8433463Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8434179Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:49.8434901Z (EngineCore_DP0 pid=10582) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:49.8435367Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8436038Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:49.8436672Z (EngineCore_DP0 pid=10582) prop = get_device_properties(device) 2025-10-10T01:57:49.8437089Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:49.8437744Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:49.8438471Z (EngineCore_DP0 pid=10582) _lazy_init() # will define _get_device_properties 2025-10-10T01:57:49.8438868Z (EngineCore_DP0 pid=10582) ^^^^^^^^^^^^ 2025-10-10T01:57:49.8439607Z (EngineCore_DP0 pid=10582) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:49.8440216Z (EngineCore_DP0 pid=10582) raise RuntimeError( 2025-10-10T01:57:49.8440901Z (EngineCore_DP0 pid=10582) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:50.2433923Z FAILED 2025-10-10T01:57:50.2564347Z models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeForCausalLM] Fork a new process to run a test 10586 2025-10-10T01:57:50.2575921Z Fork a new process to run a test 0 2025-10-10T01:57:50.2852277Z INFO 10-10 01:57:50 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GraniteMoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ibm/PowerMoE-3b'} 2025-10-10T01:57:50.6317816Z 2025-10-10T01:57:50.6318771Z config.json: 0% 0.00/928 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:57:58.5417342Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] EngineCore failed to start. 2025-10-10T01:57:58.5417879Z 2025-10-10T01:57:58.5418362Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] Traceback (most recent call last): 2025-10-10T01:57:58.5418844Z 2025-10-10T01:57:58.5419919Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:58.5420727Z 2025-10-10T01:57:58.5421302Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:58.5421709Z 2025-10-10T01:57:58.5422229Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5422533Z 2025-10-10T01:57:58.5423160Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:58.5423623Z 2025-10-10T01:57:58.5424092Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:58.5424445Z 2025-10-10T01:57:58.5425087Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:58.5425654Z 2025-10-10T01:57:58.5426018Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:58.5426597Z 2025-10-10T01:57:58.5427158Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5427494Z 2025-10-10T01:57:58.5428403Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:58.5429172Z 2025-10-10T01:57:58.5429588Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] self._init_executor() 2025-10-10T01:57:58.5429882Z 2025-10-10T01:57:58.5430453Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:58.5430974Z 2025-10-10T01:57:58.5431306Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:58.5431638Z 2025-10-10T01:57:58.5432184Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:58.5432816Z 2025-10-10T01:57:58.5433177Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:58.5433599Z 2025-10-10T01:57:58.5434000Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5434291Z 2025-10-10T01:57:58.5434810Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:58.5435273Z 2025-10-10T01:57:58.5435549Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:57:58.5435819Z 2025-10-10T01:57:58.5436071Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5436338Z 2025-10-10T01:57:58.5436857Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:58.5437335Z 2025-10-10T01:57:58.5437625Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:58.5437964Z 2025-10-10T01:57:58.5438235Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5438519Z 2025-10-10T01:57:58.5439191Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:58.5439679Z 2025-10-10T01:57:58.5439994Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:57:58.5440297Z 2025-10-10T01:57:58.5440576Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5440893Z 2025-10-10T01:57:58.5441342Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:58.5441774Z 2025-10-10T01:57:58.5442113Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:58.5442442Z 2025-10-10T01:57:58.5442736Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5443017Z 2025-10-10T01:57:58.5443363Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:57:58.5443698Z 2025-10-10T01:57:58.5444056Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:57:58.5444392Z 2025-10-10T01:57:58.5444767Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:58.5445137Z 2025-10-10T01:57:58.5445483Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:57:58.5445820Z 2025-10-10T01:57:58.5446179Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:57:58.5446534Z 2025-10-10T01:57:58.5446908Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:58.5447324Z 2025-10-10T01:57:58.5447861Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:58.5448382Z 2025-10-10T01:57:58.5448747Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:58.5449080Z 2025-10-10T01:57:58.5449605Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:58.5450080Z 2025-10-10T01:57:58.5450458Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:58.5450815Z 2025-10-10T01:57:58.5451360Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:58.5451880Z 2025-10-10T01:57:58.5452179Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:57:58.5452486Z 2025-10-10T01:57:58.5453159Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:58.5453780Z 2025-10-10T01:57:58.5454154Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:58.5454514Z 2025-10-10T01:57:58.5454770Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5455037Z 2025-10-10T01:57:58.5455608Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:58.5456131Z 2025-10-10T01:57:58.5456440Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:57:58.5456739Z 2025-10-10T01:57:58.5457020Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5457294Z 2025-10-10T01:57:58.5457895Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:58.5458443Z 2025-10-10T01:57:58.5458732Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:57:58.5459025Z 2025-10-10T01:57:58.5459286Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5459556Z 2025-10-10T01:57:58.5460147Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:58.5460685Z 2025-10-10T01:57:58.5461003Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:58.5461326Z 2025-10-10T01:57:58.5461609Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5461881Z 2025-10-10T01:57:58.5462419Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:58.5462958Z 2025-10-10T01:57:58.5463307Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:57:58.5463599Z 2025-10-10T01:57:58.5463914Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5464202Z 2025-10-10T01:57:58.5464735Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:58.5465226Z 2025-10-10T01:57:58.5465540Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:57:58.5465853Z 2025-10-10T01:57:58.5466081Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:57:58.5466332Z 2025-10-10T01:57:58.5466813Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:58.5467264Z 2025-10-10T01:57:58.5467521Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] raise RuntimeError( 2025-10-10T01:57:58.5467783Z 2025-10-10T01:57:58.5468402Z (EngineCore_DP0 pid=10644) ERROR 10-10 01:57:58 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:58.5469103Z (EngineCore_DP0 pid=10644) Process EngineCore_DP0: 2025-10-10T01:57:58.5469507Z (EngineCore_DP0 pid=10644) Traceback (most recent call last): 2025-10-10T01:57:58.5470126Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:57:58.5470649Z (EngineCore_DP0 pid=10644) self.run() 2025-10-10T01:57:58.5471195Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:57:58.5471765Z (EngineCore_DP0 pid=10644) self._target(*self._args, **self._kwargs) 2025-10-10T01:57:58.5472443Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:57:58.5473011Z (EngineCore_DP0 pid=10644) raise e 2025-10-10T01:57:58.5473607Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:57:58.5474251Z (EngineCore_DP0 pid=10644) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:57:58.5474701Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5475320Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:57:58.5475973Z (EngineCore_DP0 pid=10644) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:57:58.5476643Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:57:58.5477266Z (EngineCore_DP0 pid=10644) self.model_executor = executor_class(vllm_config) 2025-10-10T01:57:58.5477726Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5478379Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:57:58.5479020Z (EngineCore_DP0 pid=10644) self._init_executor() 2025-10-10T01:57:58.5479761Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:57:58.5480495Z (EngineCore_DP0 pid=10644) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:57:58.5481274Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:57:58.5481989Z (EngineCore_DP0 pid=10644) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:57:58.5482491Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5483141Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:57:58.5483732Z (EngineCore_DP0 pid=10644) return func(*args, **kwargs) 2025-10-10T01:57:58.5484124Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5484761Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:57:58.5485440Z (EngineCore_DP0 pid=10644) worker_class = resolve_obj_by_qualname( 2025-10-10T01:57:58.5485875Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5486544Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:57:58.5487200Z (EngineCore_DP0 pid=10644) module = importlib.import_module(module_name) 2025-10-10T01:57:58.5487643Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5488223Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:57:58.5488854Z (EngineCore_DP0 pid=10644) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:57:58.5489346Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5489846Z (EngineCore_DP0 pid=10644) File "", line 1387, in _gcd_import 2025-10-10T01:57:58.5490402Z (EngineCore_DP0 pid=10644) File "", line 1360, in _find_and_load 2025-10-10T01:57:58.5490983Z (EngineCore_DP0 pid=10644) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:57:58.5491580Z (EngineCore_DP0 pid=10644) File "", line 935, in _load_unlocked 2025-10-10T01:57:58.5492158Z (EngineCore_DP0 pid=10644) File "", line 999, in exec_module 2025-10-10T01:57:58.5492759Z (EngineCore_DP0 pid=10644) File "", line 488, in _call_with_frames_removed 2025-10-10T01:57:58.5493495Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:57:58.5494175Z (EngineCore_DP0 pid=10644) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:57:58.5494895Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:57:58.5495614Z (EngineCore_DP0 pid=10644) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:57:58.5496752Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:57:58.5497582Z (EngineCore_DP0 pid=10644) class FlashAttentionMetadataBuilder( 2025-10-10T01:57:58.5498458Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:57:58.5499346Z (EngineCore_DP0 pid=10644) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:57:58.5499830Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5500528Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:57:58.5501224Z (EngineCore_DP0 pid=10644) if not is_fa_version_supported(fa_version): 2025-10-10T01:57:58.5501679Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5502432Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:57:58.5503138Z (EngineCore_DP0 pid=10644) return _is_fa2_supported(device)[0] 2025-10-10T01:57:58.5503625Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5504349Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:57:58.5505072Z (EngineCore_DP0 pid=10644) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:57:58.5505526Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5506197Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:57:58.5506837Z (EngineCore_DP0 pid=10644) prop = get_device_properties(device) 2025-10-10T01:57:58.5507256Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:57:58.5507921Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:57:58.5508578Z (EngineCore_DP0 pid=10644) _lazy_init() # will define _get_device_properties 2025-10-10T01:57:58.5508979Z (EngineCore_DP0 pid=10644) ^^^^^^^^^^^^ 2025-10-10T01:57:58.5509566Z (EngineCore_DP0 pid=10644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:57:58.5510146Z (EngineCore_DP0 pid=10644) raise RuntimeError( 2025-10-10T01:57:58.5510838Z (EngineCore_DP0 pid=10644) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:57:58.9489026Z FAILED 2025-10-10T01:57:58.9617514Z models/test_initialization.py::test_can_initialize_large_subset[PhiForCausalLM] Fork a new process to run a test 10648 2025-10-10T01:57:58.9629531Z Fork a new process to run a test 0 2025-10-10T01:57:58.9909882Z INFO 10-10 01:57:58 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='PhiForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'microsoft/phi-2'} 2025-10-10T01:57:59.1035320Z 2025-10-10T01:57:59.1037019Z config.json: 0% 0.00/735 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:07.0595839Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:07.0597640Z 2025-10-10T01:58:07.0598087Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:07.0598477Z 2025-10-10T01:58:07.0599416Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:07.0600110Z 2025-10-10T01:58:07.0600456Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:07.0600768Z 2025-10-10T01:58:07.0601153Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0601653Z 2025-10-10T01:58:07.0602172Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:07.0602651Z 2025-10-10T01:58:07.0603000Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:07.0603325Z 2025-10-10T01:58:07.0603833Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:07.0604281Z 2025-10-10T01:58:07.0604833Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:07.0605340Z 2025-10-10T01:58:07.0605956Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0606525Z 2025-10-10T01:58:07.0607582Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:07.0608124Z 2025-10-10T01:58:07.0608424Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] self._init_executor() 2025-10-10T01:58:07.0608700Z 2025-10-10T01:58:07.0609249Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:07.0609759Z 2025-10-10T01:58:07.0610084Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:07.0610412Z 2025-10-10T01:58:07.0610971Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:07.0611476Z 2025-10-10T01:58:07.0611822Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:07.0612156Z 2025-10-10T01:58:07.0612451Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0612732Z 2025-10-10T01:58:07.0613235Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:07.0613690Z 2025-10-10T01:58:07.0613962Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:07.0614230Z 2025-10-10T01:58:07.0614480Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0614744Z 2025-10-10T01:58:07.0615255Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:07.0615871Z 2025-10-10T01:58:07.0616166Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:07.0616474Z 2025-10-10T01:58:07.0616788Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0617110Z 2025-10-10T01:58:07.0617650Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:07.0618131Z 2025-10-10T01:58:07.0618441Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:07.0618774Z 2025-10-10T01:58:07.0619046Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0619334Z 2025-10-10T01:58:07.0619783Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:07.0620203Z 2025-10-10T01:58:07.0620545Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:07.0620877Z 2025-10-10T01:58:07.0621217Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0621513Z 2025-10-10T01:58:07.0621877Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:07.0622211Z 2025-10-10T01:58:07.0622568Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:07.0622914Z 2025-10-10T01:58:07.0623291Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:07.0623657Z 2025-10-10T01:58:07.0624002Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:07.0624355Z 2025-10-10T01:58:07.0624728Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:07.0625089Z 2025-10-10T01:58:07.0625467Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:07.0625834Z 2025-10-10T01:58:07.0626342Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:07.0626812Z 2025-10-10T01:58:07.0627171Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:07.0627503Z 2025-10-10T01:58:07.0628035Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:07.0628515Z 2025-10-10T01:58:07.0628892Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:07.0629255Z 2025-10-10T01:58:07.0629802Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:07.0630370Z 2025-10-10T01:58:07.0630669Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:07.0630979Z 2025-10-10T01:58:07.0631640Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:07.0632266Z 2025-10-10T01:58:07.0632635Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:07.0633000Z 2025-10-10T01:58:07.0633257Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0633541Z 2025-10-10T01:58:07.0634102Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:07.0634625Z 2025-10-10T01:58:07.0634922Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:07.0635215Z 2025-10-10T01:58:07.0635499Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0635774Z 2025-10-10T01:58:07.0636432Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:07.0636985Z 2025-10-10T01:58:07.0637278Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:07.0637562Z 2025-10-10T01:58:07.0637826Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0638098Z 2025-10-10T01:58:07.0638690Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:07.0639355Z 2025-10-10T01:58:07.0639689Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:07.0640014Z 2025-10-10T01:58:07.0640286Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0640583Z 2025-10-10T01:58:07.0641129Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:07.0641618Z 2025-10-10T01:58:07.0641921Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:07.0642209Z 2025-10-10T01:58:07.0642485Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0642757Z 2025-10-10T01:58:07.0643287Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:07.0643766Z 2025-10-10T01:58:07.0644091Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:07.0644420Z 2025-10-10T01:58:07.0644668Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:07.0644925Z 2025-10-10T01:58:07.0645429Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:07.0645953Z 2025-10-10T01:58:07.0646215Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] raise RuntimeError( 2025-10-10T01:58:07.0646488Z 2025-10-10T01:58:07.0647094Z (EngineCore_DP0 pid=10706) ERROR 10-10 01:58:07 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:07.0647864Z (EngineCore_DP0 pid=10706) Process EngineCore_DP0: 2025-10-10T01:58:07.0648269Z (EngineCore_DP0 pid=10706) Traceback (most recent call last): 2025-10-10T01:58:07.0648877Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:07.0649395Z (EngineCore_DP0 pid=10706) self.run() 2025-10-10T01:58:07.0649941Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:07.0650510Z (EngineCore_DP0 pid=10706) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:07.0651181Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:07.0651728Z (EngineCore_DP0 pid=10706) raise e 2025-10-10T01:58:07.0652382Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:07.0653030Z (EngineCore_DP0 pid=10706) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:07.0653478Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0654103Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:07.0654745Z (EngineCore_DP0 pid=10706) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:07.0655423Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:07.0656051Z (EngineCore_DP0 pid=10706) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:07.0656509Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0657160Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:07.0657745Z (EngineCore_DP0 pid=10706) self._init_executor() 2025-10-10T01:58:07.0658412Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:07.0659119Z (EngineCore_DP0 pid=10706) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:07.0659858Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:07.0660577Z (EngineCore_DP0 pid=10706) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:07.0661080Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0661717Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:07.0662318Z (EngineCore_DP0 pid=10706) return func(*args, **kwargs) 2025-10-10T01:58:07.0662708Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0663417Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:07.0664055Z (EngineCore_DP0 pid=10706) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:07.0664524Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0665264Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:07.0665936Z (EngineCore_DP0 pid=10706) module = importlib.import_module(module_name) 2025-10-10T01:58:07.0666391Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0666975Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:07.0667597Z (EngineCore_DP0 pid=10706) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:07.0668095Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0668595Z (EngineCore_DP0 pid=10706) File "", line 1387, in _gcd_import 2025-10-10T01:58:07.0669201Z (EngineCore_DP0 pid=10706) File "", line 1360, in _find_and_load 2025-10-10T01:58:07.0669813Z (EngineCore_DP0 pid=10706) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:07.0670391Z (EngineCore_DP0 pid=10706) File "", line 935, in _load_unlocked 2025-10-10T01:58:07.0670974Z (EngineCore_DP0 pid=10706) File "", line 999, in exec_module 2025-10-10T01:58:07.0671583Z (EngineCore_DP0 pid=10706) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:07.0672316Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:07.0672998Z (EngineCore_DP0 pid=10706) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:07.0673722Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:07.0674443Z (EngineCore_DP0 pid=10706) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:07.0675225Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:07.0675896Z (EngineCore_DP0 pid=10706) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:07.0676685Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:07.0677501Z (EngineCore_DP0 pid=10706) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:07.0677977Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0678669Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:07.0679419Z (EngineCore_DP0 pid=10706) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:07.0679869Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0680619Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:07.0681417Z (EngineCore_DP0 pid=10706) return _is_fa2_supported(device)[0] 2025-10-10T01:58:07.0681850Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0682628Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:07.0683405Z (EngineCore_DP0 pid=10706) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:07.0683867Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0684532Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:07.0685172Z (EngineCore_DP0 pid=10706) prop = get_device_properties(device) 2025-10-10T01:58:07.0685606Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:07.0686276Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:07.0686946Z (EngineCore_DP0 pid=10706) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:07.0687384Z (EngineCore_DP0 pid=10706) ^^^^^^^^^^^^ 2025-10-10T01:58:07.0687990Z (EngineCore_DP0 pid=10706) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:07.0688563Z (EngineCore_DP0 pid=10706) raise RuntimeError( 2025-10-10T01:58:07.0689252Z (EngineCore_DP0 pid=10706) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:07.4695832Z FAILED 2025-10-10T01:58:07.4827547Z models/test_initialization.py::test_can_initialize_large_subset[Blip2ForConditionalGeneration] Fork a new process to run a test 10710 2025-10-10T01:58:07.4839410Z Fork a new process to run a test 0 2025-10-10T01:58:07.5111877Z INFO 10-10 01:58:07 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Blip2ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Salesforce/blip2-opt-2.7b'} 2025-10-10T01:58:07.6402349Z 2025-10-10T01:58:07.6404250Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:07.6404548Z config.json: 1.03kB [00:00, 6.23MB/s] 2025-10-10T01:58:07.7435219Z 2025-10-10T01:58:07.7436089Z preprocessor_config.json: 0% 0.00/432 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:15.7337429Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:15.7338018Z 2025-10-10T01:58:15.7338451Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:15.7338836Z 2025-10-10T01:58:15.7339529Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:15.7340127Z 2025-10-10T01:58:15.7340520Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:15.7340896Z 2025-10-10T01:58:15.7341248Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7341895Z 2025-10-10T01:58:15.7342506Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:15.7343259Z 2025-10-10T01:58:15.7343850Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:15.7344496Z 2025-10-10T01:58:15.7345153Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:15.7345732Z 2025-10-10T01:58:15.7346177Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:15.7346725Z 2025-10-10T01:58:15.7347392Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7347876Z 2025-10-10T01:58:15.7348504Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:15.7349234Z 2025-10-10T01:58:15.7349539Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] self._init_executor() 2025-10-10T01:58:15.7349956Z 2025-10-10T01:58:15.7350742Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:15.7351279Z 2025-10-10T01:58:15.7351617Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:15.7351940Z 2025-10-10T01:58:15.7352480Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:15.7352992Z 2025-10-10T01:58:15.7353344Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:15.7353685Z 2025-10-10T01:58:15.7353975Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7354268Z 2025-10-10T01:58:15.7354758Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:15.7355209Z 2025-10-10T01:58:15.7355479Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:15.7355749Z 2025-10-10T01:58:15.7356004Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7356262Z 2025-10-10T01:58:15.7356785Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:15.7357268Z 2025-10-10T01:58:15.7357554Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:15.7357846Z 2025-10-10T01:58:15.7358114Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7358388Z 2025-10-10T01:58:15.7358910Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:15.7359556Z 2025-10-10T01:58:15.7359860Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:15.7360230Z 2025-10-10T01:58:15.7360520Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7360794Z 2025-10-10T01:58:15.7361283Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:15.7361752Z 2025-10-10T01:58:15.7362139Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:15.7362495Z 2025-10-10T01:58:15.7362786Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7363069Z 2025-10-10T01:58:15.7363412Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:15.7363743Z 2025-10-10T01:58:15.7364097Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:15.7364432Z 2025-10-10T01:58:15.7364810Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:15.7365169Z 2025-10-10T01:58:15.7365558Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:15.7365890Z 2025-10-10T01:58:15.7366262Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:15.7366619Z 2025-10-10T01:58:15.7366996Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:15.7367375Z 2025-10-10T01:58:15.7367879Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:15.7368345Z 2025-10-10T01:58:15.7368702Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:15.7369050Z 2025-10-10T01:58:15.7369569Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:15.7370049Z 2025-10-10T01:58:15.7370429Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:15.7370786Z 2025-10-10T01:58:15.7371339Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:15.7371852Z 2025-10-10T01:58:15.7372163Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:15.7372465Z 2025-10-10T01:58:15.7373095Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:15.7373670Z 2025-10-10T01:58:15.7374035Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:15.7374392Z 2025-10-10T01:58:15.7374648Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7374962Z 2025-10-10T01:58:15.7375526Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:15.7376083Z 2025-10-10T01:58:15.7376386Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:15.7376725Z 2025-10-10T01:58:15.7377002Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7377274Z 2025-10-10T01:58:15.7377873Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:15.7378443Z 2025-10-10T01:58:15.7378723Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:15.7379021Z 2025-10-10T01:58:15.7379281Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7379555Z 2025-10-10T01:58:15.7380176Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:15.7380741Z 2025-10-10T01:58:15.7388776Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:15.7389192Z 2025-10-10T01:58:15.7389555Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7389848Z 2025-10-10T01:58:15.7390415Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:15.7390923Z 2025-10-10T01:58:15.7391228Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:15.7391544Z 2025-10-10T01:58:15.7391823Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7392109Z 2025-10-10T01:58:15.7392657Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:15.7393156Z 2025-10-10T01:58:15.7393474Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:15.7393784Z 2025-10-10T01:58:15.7394023Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:15.7394271Z 2025-10-10T01:58:15.7394767Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:15.7395220Z 2025-10-10T01:58:15.7395479Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] raise RuntimeError( 2025-10-10T01:58:15.7395745Z 2025-10-10T01:58:15.7396526Z (EngineCore_DP0 pid=10768) ERROR 10-10 01:58:15 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:15.7397252Z (EngineCore_DP0 pid=10768) Process EngineCore_DP0: 2025-10-10T01:58:15.7397659Z (EngineCore_DP0 pid=10768) Traceback (most recent call last): 2025-10-10T01:58:15.7398279Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:15.7398964Z (EngineCore_DP0 pid=10768) self.run() 2025-10-10T01:58:15.7399602Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:15.7400252Z (EngineCore_DP0 pid=10768) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:15.7401050Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:15.7401614Z (EngineCore_DP0 pid=10768) raise e 2025-10-10T01:58:15.7402219Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:15.7402857Z (EngineCore_DP0 pid=10768) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:15.7403307Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7403932Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:15.7404578Z (EngineCore_DP0 pid=10768) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:15.7405313Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:15.7405969Z (EngineCore_DP0 pid=10768) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:15.7406433Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7407108Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:15.7407705Z (EngineCore_DP0 pid=10768) self._init_executor() 2025-10-10T01:58:15.7408384Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:15.7409094Z (EngineCore_DP0 pid=10768) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:15.7409833Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:15.7410553Z (EngineCore_DP0 pid=10768) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:15.7411056Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7411704Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:15.7412299Z (EngineCore_DP0 pid=10768) return func(*args, **kwargs) 2025-10-10T01:58:15.7412683Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7413346Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:15.7413990Z (EngineCore_DP0 pid=10768) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:15.7414425Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7415097Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:15.7415759Z (EngineCore_DP0 pid=10768) module = importlib.import_module(module_name) 2025-10-10T01:58:15.7416204Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7416868Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:15.7417550Z (EngineCore_DP0 pid=10768) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:15.7418093Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7418599Z (EngineCore_DP0 pid=10768) File "", line 1387, in _gcd_import 2025-10-10T01:58:15.7419156Z (EngineCore_DP0 pid=10768) File "", line 1360, in _find_and_load 2025-10-10T01:58:15.7419750Z (EngineCore_DP0 pid=10768) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:15.7420338Z (EngineCore_DP0 pid=10768) File "", line 935, in _load_unlocked 2025-10-10T01:58:15.7420924Z (EngineCore_DP0 pid=10768) File "", line 999, in exec_module 2025-10-10T01:58:15.7421548Z (EngineCore_DP0 pid=10768) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:15.7422285Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:15.7423019Z (EngineCore_DP0 pid=10768) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:15.7423767Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:15.7424498Z (EngineCore_DP0 pid=10768) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:15.7425294Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:15.7425986Z (EngineCore_DP0 pid=10768) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:15.7426777Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:15.7427602Z (EngineCore_DP0 pid=10768) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:15.7428089Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7428791Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:15.7429478Z (EngineCore_DP0 pid=10768) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:15.7429920Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7430668Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:15.7431384Z (EngineCore_DP0 pid=10768) return _is_fa2_supported(device)[0] 2025-10-10T01:58:15.7431808Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7432531Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:15.7433244Z (EngineCore_DP0 pid=10768) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:15.7433704Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7434377Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:15.7435070Z (EngineCore_DP0 pid=10768) prop = get_device_properties(device) 2025-10-10T01:58:15.7435530Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:15.7436242Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:15.7436912Z (EngineCore_DP0 pid=10768) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:15.7437316Z (EngineCore_DP0 pid=10768) ^^^^^^^^^^^^ 2025-10-10T01:58:15.7437900Z (EngineCore_DP0 pid=10768) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:15.7438467Z (EngineCore_DP0 pid=10768) raise RuntimeError( 2025-10-10T01:58:15.7439237Z (EngineCore_DP0 pid=10768) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:16.1442749Z FAILED 2025-10-10T01:58:16.1574021Z models/test_initialization.py::test_can_initialize_large_subset[MixtralForCausalLM] Fork a new process to run a test 10772 2025-10-10T01:58:16.1585646Z Fork a new process to run a test 0 2025-10-10T01:58:16.1862377Z INFO 10-10 01:58:16 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MixtralForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1'} 2025-10-10T01:58:16.2818445Z 2025-10-10T01:58:16.2819842Z config.json: 0% 0.00/720 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:23.9030373Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:23.9031023Z 2025-10-10T01:58:23.9031660Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:23.9032303Z 2025-10-10T01:58:23.9033152Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:23.9033780Z 2025-10-10T01:58:23.9034223Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:23.9034611Z 2025-10-10T01:58:23.9034994Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9035326Z 2025-10-10T01:58:23.9035818Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:23.9036275Z 2025-10-10T01:58:23.9036622Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:23.9037164Z 2025-10-10T01:58:23.9038021Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:23.9038493Z 2025-10-10T01:58:23.9038829Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:23.9039282Z 2025-10-10T01:58:23.9039577Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9039855Z 2025-10-10T01:58:23.9040492Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:23.9041325Z 2025-10-10T01:58:23.9042081Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] self._init_executor() 2025-10-10T01:58:23.9042553Z 2025-10-10T01:58:23.9043630Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:23.9044669Z 2025-10-10T01:58:23.9045267Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:23.9045842Z 2025-10-10T01:58:23.9046464Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:23.9046981Z 2025-10-10T01:58:23.9047335Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:23.9047682Z 2025-10-10T01:58:23.9047975Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9048258Z 2025-10-10T01:58:23.9048757Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:23.9049225Z 2025-10-10T01:58:23.9049614Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:23.9049893Z 2025-10-10T01:58:23.9050156Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9050414Z 2025-10-10T01:58:23.9050930Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:23.9051422Z 2025-10-10T01:58:23.9051716Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:23.9052011Z 2025-10-10T01:58:23.9052282Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9052560Z 2025-10-10T01:58:23.9053086Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:23.9053575Z 2025-10-10T01:58:23.9053877Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:23.9054175Z 2025-10-10T01:58:23.9054455Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9054760Z 2025-10-10T01:58:23.9055229Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:23.9055673Z 2025-10-10T01:58:23.9056034Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:23.9056377Z 2025-10-10T01:58:23.9056670Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9056949Z 2025-10-10T01:58:23.9057311Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:23.9057641Z 2025-10-10T01:58:23.9057996Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:23.9058332Z 2025-10-10T01:58:23.9058773Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:23.9059134Z 2025-10-10T01:58:23.9059514Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:23.9059901Z 2025-10-10T01:58:23.9060282Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:23.9060641Z 2025-10-10T01:58:23.9061016Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:23.9061382Z 2025-10-10T01:58:23.9061883Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:23.9062356Z 2025-10-10T01:58:23.9062707Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:23.9063045Z 2025-10-10T01:58:23.9063569Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:23.9064050Z 2025-10-10T01:58:23.9064469Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:23.9064837Z 2025-10-10T01:58:23.9065390Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:23.9065905Z 2025-10-10T01:58:23.9066206Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:23.9066517Z 2025-10-10T01:58:23.9067147Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:23.9067728Z 2025-10-10T01:58:23.9068095Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:23.9068475Z 2025-10-10T01:58:23.9068736Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9069018Z 2025-10-10T01:58:23.9069606Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:23.9070141Z 2025-10-10T01:58:23.9070441Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:23.9070736Z 2025-10-10T01:58:23.9071015Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9071289Z 2025-10-10T01:58:23.9071896Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:23.9072444Z 2025-10-10T01:58:23.9072733Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:23.9073017Z 2025-10-10T01:58:23.9073278Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9073552Z 2025-10-10T01:58:23.9074186Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:23.9074739Z 2025-10-10T01:58:23.9075099Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:23.9075466Z 2025-10-10T01:58:23.9075755Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9076040Z 2025-10-10T01:58:23.9076564Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:23.9077047Z 2025-10-10T01:58:23.9077340Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:23.9077630Z 2025-10-10T01:58:23.9077909Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9078184Z 2025-10-10T01:58:23.9078713Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:23.9079297Z 2025-10-10T01:58:23.9079694Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:23.9080009Z 2025-10-10T01:58:23.9080239Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:23.9080492Z 2025-10-10T01:58:23.9080974Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:23.9081427Z 2025-10-10T01:58:23.9081675Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] raise RuntimeError( 2025-10-10T01:58:23.9081939Z 2025-10-10T01:58:23.9082504Z (EngineCore_DP0 pid=10850) ERROR 10-10 01:58:23 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:23.9083201Z (EngineCore_DP0 pid=10850) Process EngineCore_DP0: 2025-10-10T01:58:23.9083601Z (EngineCore_DP0 pid=10850) Traceback (most recent call last): 2025-10-10T01:58:23.9084216Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:23.9084735Z (EngineCore_DP0 pid=10850) self.run() 2025-10-10T01:58:23.9085269Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:23.9085847Z (EngineCore_DP0 pid=10850) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:23.9086517Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:23.9087063Z (EngineCore_DP0 pid=10850) raise e 2025-10-10T01:58:23.9087663Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:23.9088295Z (EngineCore_DP0 pid=10850) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:23.9088737Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9089362Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:23.9090003Z (EngineCore_DP0 pid=10850) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:23.9090732Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:23.9091397Z (EngineCore_DP0 pid=10850) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:23.9091907Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9092560Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:23.9093139Z (EngineCore_DP0 pid=10850) self._init_executor() 2025-10-10T01:58:23.9093804Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:23.9094506Z (EngineCore_DP0 pid=10850) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:23.9095235Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:23.9095954Z (EngineCore_DP0 pid=10850) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:23.9096741Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9097405Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:23.9097994Z (EngineCore_DP0 pid=10850) return func(*args, **kwargs) 2025-10-10T01:58:23.9098383Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9099033Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:23.9099667Z (EngineCore_DP0 pid=10850) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:23.9100092Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9100763Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:23.9101423Z (EngineCore_DP0 pid=10850) module = importlib.import_module(module_name) 2025-10-10T01:58:23.9101873Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9102458Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:23.9103076Z (EngineCore_DP0 pid=10850) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:23.9103574Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9104074Z (EngineCore_DP0 pid=10850) File "", line 1387, in _gcd_import 2025-10-10T01:58:23.9104632Z (EngineCore_DP0 pid=10850) File "", line 1360, in _find_and_load 2025-10-10T01:58:23.9105224Z (EngineCore_DP0 pid=10850) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:23.9105800Z (EngineCore_DP0 pid=10850) File "", line 935, in _load_unlocked 2025-10-10T01:58:23.9106379Z (EngineCore_DP0 pid=10850) File "", line 999, in exec_module 2025-10-10T01:58:23.9106981Z (EngineCore_DP0 pid=10850) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:23.9107806Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:23.9108481Z (EngineCore_DP0 pid=10850) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:23.9109265Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:23.9110058Z (EngineCore_DP0 pid=10850) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:23.9110839Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:23.9111508Z (EngineCore_DP0 pid=10850) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:23.9112298Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:23.9113115Z (EngineCore_DP0 pid=10850) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:23.9113590Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9114338Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:23.9115030Z (EngineCore_DP0 pid=10850) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:23.9115470Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9116228Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:23.9116938Z (EngineCore_DP0 pid=10850) return _is_fa2_supported(device)[0] 2025-10-10T01:58:23.9117351Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9118069Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:23.9118793Z (EngineCore_DP0 pid=10850) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:23.9119367Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9120043Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:23.9120680Z (EngineCore_DP0 pid=10850) prop = get_device_properties(device) 2025-10-10T01:58:23.9121101Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:23.9121763Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:23.9122425Z (EngineCore_DP0 pid=10850) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:23.9122822Z (EngineCore_DP0 pid=10850) ^^^^^^^^^^^^ 2025-10-10T01:58:23.9123401Z (EngineCore_DP0 pid=10850) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:23.9123967Z (EngineCore_DP0 pid=10850) raise RuntimeError( 2025-10-10T01:58:23.9124661Z (EngineCore_DP0 pid=10850) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:24.3123409Z FAILED 2025-10-10T01:58:24.3253851Z models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5_VLMoeForConditionalGeneration] Fork a new process to run a test 10854 2025-10-10T01:58:24.3265089Z Fork a new process to run a test 0 2025-10-10T01:58:24.3542423Z INFO 10-10 01:58:24 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Ernie4_5_VLMoeForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'baidu/ERNIE-4.5-VL-28B-A3B-PT'} 2025-10-10T01:58:24.4588849Z 2025-10-10T01:58:24.4590807Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:24.4591284Z config.json: 2.01kB [00:00, 10.8MB/s] 2025-10-10T01:58:24.6406916Z 2025-10-10T01:58:24.6409674Z configuration_ernie4_5_vl.py: 0.00B [00:00, ?B/s] 2025-10-10T01:58:24.6410324Z configuration_ernie4_5_vl.py: 28.3kB [00:00, 72.5MB/s] 2025-10-10T01:58:24.6537770Z A new version of the following files was downloaded from https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT: 2025-10-10T01:58:24.6538688Z - configuration_ernie4_5_vl.py 2025-10-10T01:58:24.6539331Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:58:24.6858303Z INFO 10-10 01:58:24 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:58:24.8687736Z 2025-10-10T01:58:24.8688777Z preprocessor_config.json: 0% 0.00/477 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:36.2679730Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:36.2680401Z 2025-10-10T01:58:36.2680832Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:36.2681194Z 2025-10-10T01:58:36.2681859Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:36.2682459Z 2025-10-10T01:58:36.2682844Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:36.2683242Z 2025-10-10T01:58:36.2683594Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2683943Z 2025-10-10T01:58:36.2684548Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:36.2685110Z 2025-10-10T01:58:36.2685776Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:36.2686194Z 2025-10-10T01:58:36.2686770Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:36.2687214Z 2025-10-10T01:58:36.2687527Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:36.2688008Z 2025-10-10T01:58:36.2688300Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2688683Z 2025-10-10T01:58:36.2689398Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:36.2690102Z 2025-10-10T01:58:36.2690438Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] self._init_executor() 2025-10-10T01:58:36.2690787Z 2025-10-10T01:58:36.2691664Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:36.2692445Z 2025-10-10T01:58:36.2692807Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:36.2693130Z 2025-10-10T01:58:36.2693671Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:36.2694173Z 2025-10-10T01:58:36.2694624Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:36.2694969Z 2025-10-10T01:58:36.2695266Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2695563Z 2025-10-10T01:58:36.2696270Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:36.2696744Z 2025-10-10T01:58:36.2697023Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:36.2697295Z 2025-10-10T01:58:36.2697551Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2697812Z 2025-10-10T01:58:36.2698333Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:36.2698821Z 2025-10-10T01:58:36.2699115Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:36.2699408Z 2025-10-10T01:58:36.2699675Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2699953Z 2025-10-10T01:58:36.2700480Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:36.2700975Z 2025-10-10T01:58:36.2701278Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:36.2701576Z 2025-10-10T01:58:36.2701855Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2702130Z 2025-10-10T01:58:36.2702578Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:36.2702990Z 2025-10-10T01:58:36.2703335Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:36.2703663Z 2025-10-10T01:58:36.2704048Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2704342Z 2025-10-10T01:58:36.2704744Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:36.2705143Z 2025-10-10T01:58:36.2705502Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:36.2705845Z 2025-10-10T01:58:36.2706232Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:36.2706599Z 2025-10-10T01:58:36.2706942Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:36.2707275Z 2025-10-10T01:58:36.2707641Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:36.2707990Z 2025-10-10T01:58:36.2708369Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:36.2708741Z 2025-10-10T01:58:36.2709323Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:36.2709795Z 2025-10-10T01:58:36.2710150Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:36.2710492Z 2025-10-10T01:58:36.2711022Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:36.2711511Z 2025-10-10T01:58:36.2711884Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:36.2712249Z 2025-10-10T01:58:36.2712799Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:36.2713311Z 2025-10-10T01:58:36.2713606Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:36.2713899Z 2025-10-10T01:58:36.2714530Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:36.2715114Z 2025-10-10T01:58:36.2715492Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:36.2715841Z 2025-10-10T01:58:36.2716107Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2716368Z 2025-10-10T01:58:36.2716947Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:36.2717463Z 2025-10-10T01:58:36.2717759Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:36.2718057Z 2025-10-10T01:58:36.2718329Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2718607Z 2025-10-10T01:58:36.2719328Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:36.2719951Z 2025-10-10T01:58:36.2720277Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:36.2720641Z 2025-10-10T01:58:36.2720916Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2721189Z 2025-10-10T01:58:36.2721768Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:36.2722303Z 2025-10-10T01:58:36.2722622Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:36.2722955Z 2025-10-10T01:58:36.2723231Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2723511Z 2025-10-10T01:58:36.2724038Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:36.2724531Z 2025-10-10T01:58:36.2724855Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:36.2725143Z 2025-10-10T01:58:36.2725421Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2725694Z 2025-10-10T01:58:36.2726225Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:36.2726709Z 2025-10-10T01:58:36.2727024Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:36.2727333Z 2025-10-10T01:58:36.2727564Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:36.2727815Z 2025-10-10T01:58:36.2728300Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:36.2728755Z 2025-10-10T01:58:36.2729002Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] raise RuntimeError( 2025-10-10T01:58:36.2729263Z 2025-10-10T01:58:36.2729826Z (EngineCore_DP0 pid=10933) ERROR 10-10 01:58:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:36.2730518Z (EngineCore_DP0 pid=10933) Process EngineCore_DP0: 2025-10-10T01:58:36.2730933Z (EngineCore_DP0 pid=10933) Traceback (most recent call last): 2025-10-10T01:58:36.2731544Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:36.2732062Z (EngineCore_DP0 pid=10933) self.run() 2025-10-10T01:58:36.2732595Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:36.2733181Z (EngineCore_DP0 pid=10933) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:36.2733850Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:36.2734401Z (EngineCore_DP0 pid=10933) raise e 2025-10-10T01:58:36.2734989Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:36.2735687Z (EngineCore_DP0 pid=10933) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:36.2736256Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2737048Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:36.2737805Z (EngineCore_DP0 pid=10933) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:36.2738479Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:36.2739091Z (EngineCore_DP0 pid=10933) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:36.2739560Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2740215Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:36.2740811Z (EngineCore_DP0 pid=10933) self._init_executor() 2025-10-10T01:58:36.2741522Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:36.2742216Z (EngineCore_DP0 pid=10933) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:36.2742937Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:36.2743652Z (EngineCore_DP0 pid=10933) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:36.2744152Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2744793Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:36.2745383Z (EngineCore_DP0 pid=10933) return func(*args, **kwargs) 2025-10-10T01:58:36.2745790Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2746558Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:36.2747312Z (EngineCore_DP0 pid=10933) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:36.2747783Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2748458Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:36.2749114Z (EngineCore_DP0 pid=10933) module = importlib.import_module(module_name) 2025-10-10T01:58:36.2749559Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2750148Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:36.2750793Z (EngineCore_DP0 pid=10933) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:36.2751289Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2751783Z (EngineCore_DP0 pid=10933) File "", line 1387, in _gcd_import 2025-10-10T01:58:36.2752338Z (EngineCore_DP0 pid=10933) File "", line 1360, in _find_and_load 2025-10-10T01:58:36.2752976Z (EngineCore_DP0 pid=10933) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:36.2753557Z (EngineCore_DP0 pid=10933) File "", line 935, in _load_unlocked 2025-10-10T01:58:36.2754167Z (EngineCore_DP0 pid=10933) File "", line 999, in exec_module 2025-10-10T01:58:36.2754805Z (EngineCore_DP0 pid=10933) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:36.2755539Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:36.2756216Z (EngineCore_DP0 pid=10933) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:36.2756937Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:36.2757656Z (EngineCore_DP0 pid=10933) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:36.2758445Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:36.2759179Z (EngineCore_DP0 pid=10933) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:36.2760021Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:36.2760839Z (EngineCore_DP0 pid=10933) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:36.2761321Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2762014Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:36.2762704Z (EngineCore_DP0 pid=10933) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:36.2763148Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2763893Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:36.2764593Z (EngineCore_DP0 pid=10933) return _is_fa2_supported(device)[0] 2025-10-10T01:58:36.2765002Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2765711Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:36.2766421Z (EngineCore_DP0 pid=10933) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:36.2766878Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2767547Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:36.2768186Z (EngineCore_DP0 pid=10933) prop = get_device_properties(device) 2025-10-10T01:58:36.2768604Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:36.2769282Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:36.2769959Z (EngineCore_DP0 pid=10933) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:36.2770358Z (EngineCore_DP0 pid=10933) ^^^^^^^^^^^^ 2025-10-10T01:58:36.2771007Z (EngineCore_DP0 pid=10933) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:36.2771577Z (EngineCore_DP0 pid=10933) raise RuntimeError( 2025-10-10T01:58:36.2772303Z (EngineCore_DP0 pid=10933) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:36.6824773Z FAILED 2025-10-10T01:58:36.6954679Z models/test_initialization.py::test_can_initialize_large_subset[TransformersForSequenceClassification] Fork a new process to run a test 10937 2025-10-10T01:58:36.6966514Z Fork a new process to run a test 0 2025-10-10T01:58:36.6970352Z `transformers==4.56.2` installed, but `transformers>=4.57.0.dev0` is required to run this model. 2025-10-10T01:58:37.0047925Z PASSED 2025-10-10T01:58:37.0177989Z models/test_initialization.py::test_can_initialize_large_subset[Terratorch] Fork a new process to run a test 10938 2025-10-10T01:58:37.0188976Z Fork a new process to run a test 0 2025-10-10T01:58:37.0460722Z INFO 10-10 01:58:37 [utils.py:233] non-default args: {'skip_tokenizer_init': True, 'load_format': 'dummy', 'dtype': torch.float16, 'gpu_memory_utilization': 0.8, 'max_num_seqs': 32, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Terratorch', exist_overrides={}, use_original_num_layers=False), 'enforce_eager': True, 'model_impl': 'vllm', 'model': 'ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11'} 2025-10-10T01:58:37.5234433Z INFO 10-10 01:58:37 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:58:37.5235100Z INFO 10-10 01:58:37 [model.py:551] Resolved architecture: Terratorch 2025-10-10T01:58:37.5235500Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:58:37.7883580Z ERROR 10-10 01:58:37 [config.py:278] Error retrieving safetensors: 'ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files., retrying 1 of 2 2025-10-10T01:58:39.8612918Z ERROR 10-10 01:58:39 [config.py:276] Error retrieving safetensors: 'ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files. 2025-10-10T01:58:39.8613999Z INFO 10-10 01:58:39 [model.py:1765] Downcasting torch.float32 to torch.float16. 2025-10-10T01:58:39.8615370Z WARNING 10-10 01:58:39 [model.py:1867] The model's config.json does not contain any of the following keys to determine the original maximum length of the model: ['max_position_embeddings', 'n_positions', 'max_seq_len', 'seq_length', 'model_max_length', 'max_target_positions', 'max_sequence_length', 'max_seq_length', 'seq_len']. Assuming the model's maximum length is 2048. 2025-10-10T01:58:39.8616639Z INFO 10-10 01:58:39 [model.py:1545] Using max model len 2048 2025-10-10T01:58:39.9105102Z INFO 10-10 01:58:39 [arg_utils.py:1580] (Disabling) chunked prefill by default 2025-10-10T01:58:39.9105589Z INFO 10-10 01:58:39 [arg_utils.py:1583] (Disabling) prefix caching by default 2025-10-10T01:58:40.0601355Z INFO 10-10 01:58:40 [vllm.py:345] Cudagraph is disabled under eager mode 2025-10-10T01:58:40.0601968Z INFO 10-10 01:58:40 [vllm.py:404] Only "last" pooling supports chunked prefill and prefix caching; disabling both. 2025-10-10T01:58:43.5816318Z (EngineCore_DP0 pid=10946) INFO 10-10 01:58:43 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:58:43.5904778Z (EngineCore_DP0 pid=10946) INFO 10-10 01:58:43 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11', speculative_config=None, tokenizer='ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11', skip_tokenizer_init=True, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11, enable_prefix_caching=False, chunked_prefill_enabled=False, pooler_config=PoolerConfig(pooling_type='All', normalize=None, dimensions=None, enable_chunked_processing=None, max_embed_len=None, activation=None, logit_bias=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={'level': 0, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': False, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 0, 'local_cache_dir': None} 2025-10-10T01:58:43.7140071Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:43.7140575Z 2025-10-10T01:58:43.7140948Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:43.7141335Z 2025-10-10T01:58:43.7142007Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:43.7142638Z 2025-10-10T01:58:43.7143031Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:43.7143378Z 2025-10-10T01:58:43.7143666Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7143960Z 2025-10-10T01:58:43.7144445Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:43.7144912Z 2025-10-10T01:58:43.7145261Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:43.7145584Z 2025-10-10T01:58:43.7146055Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:43.7146498Z 2025-10-10T01:58:43.7146810Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:43.7147117Z 2025-10-10T01:58:43.7147401Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7147678Z 2025-10-10T01:58:43.7148194Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:43.7148659Z 2025-10-10T01:58:43.7148923Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] self._init_executor() 2025-10-10T01:58:43.7149180Z 2025-10-10T01:58:43.7149715Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:43.7150691Z 2025-10-10T01:58:43.7151500Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:43.7152139Z 2025-10-10T01:58:43.7153332Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:43.7154081Z 2025-10-10T01:58:43.7154473Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:43.7154835Z 2025-10-10T01:58:43.7155145Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7155441Z 2025-10-10T01:58:43.7155949Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:43.7156410Z 2025-10-10T01:58:43.7156689Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:43.7156963Z 2025-10-10T01:58:43.7157357Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7157643Z 2025-10-10T01:58:43.7158169Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:43.7158655Z 2025-10-10T01:58:43.7158949Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:43.7159392Z 2025-10-10T01:58:43.7159678Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7159970Z 2025-10-10T01:58:43.7160519Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:43.7161021Z 2025-10-10T01:58:43.7161330Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:43.7161634Z 2025-10-10T01:58:43.7162064Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7162494Z 2025-10-10T01:58:43.7163225Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:43.7163874Z 2025-10-10T01:58:43.7164430Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:43.7165000Z 2025-10-10T01:58:43.7165538Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7166010Z 2025-10-10T01:58:43.7166550Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:43.7167078Z 2025-10-10T01:58:43.7167645Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:43.7168001Z 2025-10-10T01:58:43.7168386Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:43.7168753Z 2025-10-10T01:58:43.7169194Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:43.7169531Z 2025-10-10T01:58:43.7169960Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:43.7170363Z 2025-10-10T01:58:43.7170753Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:43.7171116Z 2025-10-10T01:58:43.7171626Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:43.7172092Z 2025-10-10T01:58:43.7172444Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:43.7172797Z 2025-10-10T01:58:43.7173331Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:43.7173842Z 2025-10-10T01:58:43.7174218Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:43.7174600Z 2025-10-10T01:58:43.7175201Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:43.7175732Z 2025-10-10T01:58:43.7176039Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:43.7176343Z 2025-10-10T01:58:43.7176984Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:43.7177580Z 2025-10-10T01:58:43.7177958Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:43.7178317Z 2025-10-10T01:58:43.7178595Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7178861Z 2025-10-10T01:58:43.7179433Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:43.7179949Z 2025-10-10T01:58:43.7180245Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:43.7180545Z 2025-10-10T01:58:43.7180822Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7181104Z 2025-10-10T01:58:43.7181704Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:43.7182263Z 2025-10-10T01:58:43.7182548Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:43.7182839Z 2025-10-10T01:58:43.7183097Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7183366Z 2025-10-10T01:58:43.7183956Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:43.7184543Z 2025-10-10T01:58:43.7184875Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:43.7185190Z 2025-10-10T01:58:43.7185525Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7185838Z 2025-10-10T01:58:43.7186374Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:43.7186870Z 2025-10-10T01:58:43.7187152Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:43.7187448Z 2025-10-10T01:58:43.7187718Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7188002Z 2025-10-10T01:58:43.7188531Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:43.7189023Z 2025-10-10T01:58:43.7189341Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:43.7189648Z 2025-10-10T01:58:43.7189949Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:43.7190199Z 2025-10-10T01:58:43.7190690Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:43.7191144Z 2025-10-10T01:58:43.7191398Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] raise RuntimeError( 2025-10-10T01:58:43.7191659Z 2025-10-10T01:58:43.7192227Z (EngineCore_DP0 pid=10946) ERROR 10-10 01:58:43 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:43.7192934Z (EngineCore_DP0 pid=10946) Process EngineCore_DP0: 2025-10-10T01:58:43.7193381Z (EngineCore_DP0 pid=10946) Traceback (most recent call last): 2025-10-10T01:58:43.7194003Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:43.7194523Z (EngineCore_DP0 pid=10946) self.run() 2025-10-10T01:58:43.7195054Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:43.7195640Z (EngineCore_DP0 pid=10946) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:43.7196683Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:43.7197271Z (EngineCore_DP0 pid=10946) raise e 2025-10-10T01:58:43.7197873Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:43.7198517Z (EngineCore_DP0 pid=10946) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:43.7198966Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7199685Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:43.7200339Z (EngineCore_DP0 pid=10946) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:43.7201006Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:43.7201766Z (EngineCore_DP0 pid=10946) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:43.7202230Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7202961Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:43.7203647Z (EngineCore_DP0 pid=10946) self._init_executor() 2025-10-10T01:58:43.7204316Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:43.7205012Z (EngineCore_DP0 pid=10946) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:43.7205739Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:43.7206466Z (EngineCore_DP0 pid=10946) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:43.7206970Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7207629Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:43.7208309Z (EngineCore_DP0 pid=10946) return func(*args, **kwargs) 2025-10-10T01:58:43.7208725Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7209370Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:43.7210001Z (EngineCore_DP0 pid=10946) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:43.7210436Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7211127Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:43.7211792Z (EngineCore_DP0 pid=10946) module = importlib.import_module(module_name) 2025-10-10T01:58:43.7212240Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7212831Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:43.7213459Z (EngineCore_DP0 pid=10946) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:43.7213965Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7214469Z (EngineCore_DP0 pid=10946) File "", line 1387, in _gcd_import 2025-10-10T01:58:43.7215031Z (EngineCore_DP0 pid=10946) File "", line 1360, in _find_and_load 2025-10-10T01:58:43.7215631Z (EngineCore_DP0 pid=10946) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:43.7216217Z (EngineCore_DP0 pid=10946) File "", line 935, in _load_unlocked 2025-10-10T01:58:43.7216794Z (EngineCore_DP0 pid=10946) File "", line 999, in exec_module 2025-10-10T01:58:43.7217402Z (EngineCore_DP0 pid=10946) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:43.7218146Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:43.7218836Z (EngineCore_DP0 pid=10946) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:43.7219628Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:43.7220403Z (EngineCore_DP0 pid=10946) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:43.7221248Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:43.7221925Z (EngineCore_DP0 pid=10946) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:43.7222730Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:43.7223547Z (EngineCore_DP0 pid=10946) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:43.7224052Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7224754Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:43.7225445Z (EngineCore_DP0 pid=10946) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:43.7225942Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7226696Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:43.7227415Z (EngineCore_DP0 pid=10946) return _is_fa2_supported(device)[0] 2025-10-10T01:58:43.7227831Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7228551Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:43.7229282Z (EngineCore_DP0 pid=10946) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:43.7229746Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7230418Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:43.7231053Z (EngineCore_DP0 pid=10946) prop = get_device_properties(device) 2025-10-10T01:58:43.7231470Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:43.7232141Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:43.7232810Z (EngineCore_DP0 pid=10946) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:43.7233206Z (EngineCore_DP0 pid=10946) ^^^^^^^^^^^^ 2025-10-10T01:58:43.7233801Z (EngineCore_DP0 pid=10946) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:43.7234373Z (EngineCore_DP0 pid=10946) raise RuntimeError( 2025-10-10T01:58:43.7235080Z (EngineCore_DP0 pid=10946) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:44.1902673Z FAILED 2025-10-10T01:58:44.2032109Z models/test_initialization.py::test_can_initialize_large_subset[MiniMaxText01ForCausalLM] Fork a new process to run a test 10950 2025-10-10T01:58:44.2044016Z Fork a new process to run a test 0 2025-10-10T01:58:44.2309711Z INFO 10-10 01:58:44 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'revision': 'a59aa9cbc53b9fb8742ca4e9e1531b9802b6fdc3', 'hf_overrides': functools.partial(, model_arch='MiniMaxText01ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'MiniMaxAI/MiniMax-Text-01'} 2025-10-10T01:58:44.3935436Z 2025-10-10T01:58:44.3937235Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:44.3937567Z config.json: 1.89kB [00:00, 10.4MB/s] 2025-10-10T01:58:44.4477932Z 2025-10-10T01:58:44.4479888Z configuration_minimax_text_01.py: 0.00B [00:00, ?B/s] 2025-10-10T01:58:44.4480313Z configuration_minimax_text_01.py: 7.36kB [00:00, 52.1MB/s] 2025-10-10T01:58:44.4575594Z You are using a model of type minimax_text_01 to instantiate a model of type MiniMaxText01. This is not supported for all configurations of models and can yield errors. 2025-10-10T01:58:44.7070153Z INFO 10-10 01:58:44 [model.py:551] Resolved architecture: MiniMaxText01ForCausalLM 2025-10-10T01:58:44.7070683Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:58:44.8244972Z 2025-10-10T01:58:44.8294843Z model.safetensors.index.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:44.8295256Z model.safetensors.index.json: 823kB [00:00, 161MB/s] 2025-10-10T01:58:44.8569810Z 2025-10-10T01:58:45.3241271Z Parse safetensors files: 0% 0/413 [00:00= mamba page size. 2025-10-10T01:58:46.5930178Z 2025-10-10T01:58:46.5931905Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:46.5932278Z tokenizer_config.json: 1.39kB [00:00, 13.2MB/s] 2025-10-10T01:58:46.6891420Z 2025-10-10T01:58:46.7112423Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:46.7112739Z vocab.json: 4.71MB [00:00, 213MB/s] 2025-10-10T01:58:46.7857143Z 2025-10-10T01:58:46.7982309Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T01:58:46.7982581Z merges.txt: 2.41MB [00:00, 192MB/s] 2025-10-10T01:58:46.9294606Z 2025-10-10T01:58:46.9676900Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:46.9677241Z tokenizer.json: 9.72MB [00:00, 255MB/s] 2025-10-10T01:58:47.6792533Z You are using a model of type minimax_text_01 to instantiate a model of type MiniMaxText01. This is not supported for all configurations of models and can yield errors. 2025-10-10T01:58:47.7002975Z (EngineCore_DP0 pid=10978) INFO 10-10 01:58:47 [core.py:648] Waiting for init message from front-end. 2025-10-10T01:58:47.7077170Z (EngineCore_DP0 pid=10978) INFO 10-10 01:58:47 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='MiniMaxAI/MiniMax-Text-01', speculative_config=None, tokenizer='MiniMaxAI/MiniMax-Text-01', skip_tokenizer_init=False, tokenizer_mode=auto, revision=a59aa9cbc53b9fb8742ca4e9e1531b9802b6fdc3, tokenizer_revision=a59aa9cbc53b9fb8742ca4e9e1531b9802b6fdc3, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=10240000, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=MiniMaxAI/MiniMax-Text-01, enable_prefix_caching=False, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:47.8132401Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:47.8133117Z 2025-10-10T01:58:47.8133529Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:47.8133911Z 2025-10-10T01:58:47.8134771Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:47.8135429Z 2025-10-10T01:58:47.8135832Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:47.8136228Z 2025-10-10T01:58:47.8136581Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8136942Z 2025-10-10T01:58:47.8137433Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:47.8137888Z 2025-10-10T01:58:47.8138267Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:47.8138609Z 2025-10-10T01:58:47.8139107Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:47.8139578Z 2025-10-10T01:58:47.8140000Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:47.8140327Z 2025-10-10T01:58:47.8140614Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8140888Z 2025-10-10T01:58:47.8141404Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:47.8142088Z 2025-10-10T01:58:47.8142372Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] self._init_executor() 2025-10-10T01:58:47.8142638Z 2025-10-10T01:58:47.8143284Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:47.8144122Z 2025-10-10T01:58:47.8144499Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:47.8144869Z 2025-10-10T01:58:47.8145751Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:47.8146294Z 2025-10-10T01:58:47.8146768Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:47.8147210Z 2025-10-10T01:58:47.8147629Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8147937Z 2025-10-10T01:58:47.8148551Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:47.8149021Z 2025-10-10T01:58:47.8149302Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:47.8149578Z 2025-10-10T01:58:47.8149839Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8150100Z 2025-10-10T01:58:47.8150645Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:47.8151131Z 2025-10-10T01:58:47.8151422Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:47.8151720Z 2025-10-10T01:58:47.8151990Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8152268Z 2025-10-10T01:58:47.8152790Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:47.8153280Z 2025-10-10T01:58:47.8153591Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:47.8153903Z 2025-10-10T01:58:47.8154181Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8154453Z 2025-10-10T01:58:47.8154924Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:47.8155348Z 2025-10-10T01:58:47.8155699Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:47.8156031Z 2025-10-10T01:58:47.8156324Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8156607Z 2025-10-10T01:58:47.8156954Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:47.8157289Z 2025-10-10T01:58:47.8157701Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:47.8158044Z 2025-10-10T01:58:47.8158470Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:47.8158890Z 2025-10-10T01:58:47.8159383Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:47.8159726Z 2025-10-10T01:58:47.8160093Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:47.8160446Z 2025-10-10T01:58:47.8160824Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:47.8161197Z 2025-10-10T01:58:47.8161728Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:47.8162202Z 2025-10-10T01:58:47.8162564Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:47.8162905Z 2025-10-10T01:58:47.8163483Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:47.8163979Z 2025-10-10T01:58:47.8164357Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:47.8164723Z 2025-10-10T01:58:47.8165272Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:47.8165790Z 2025-10-10T01:58:47.8166086Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:47.8166392Z 2025-10-10T01:58:47.8167036Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:47.8167630Z 2025-10-10T01:58:47.8168000Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:47.8168354Z 2025-10-10T01:58:47.8168617Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8168881Z 2025-10-10T01:58:47.8169451Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:47.8169969Z 2025-10-10T01:58:47.8170287Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:47.8170586Z 2025-10-10T01:58:47.8170861Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8171142Z 2025-10-10T01:58:47.8171742Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:47.8172318Z 2025-10-10T01:58:47.8172614Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:47.8172911Z 2025-10-10T01:58:47.8173240Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8173516Z 2025-10-10T01:58:47.8174185Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:47.8174763Z 2025-10-10T01:58:47.8175098Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:47.8175413Z 2025-10-10T01:58:47.8175692Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8175968Z 2025-10-10T01:58:47.8176503Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:47.8176999Z 2025-10-10T01:58:47.8177296Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:47.8177591Z 2025-10-10T01:58:47.8177864Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8178142Z 2025-10-10T01:58:47.8178714Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:47.8179212Z 2025-10-10T01:58:47.8179529Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:47.8179832Z 2025-10-10T01:58:47.8180066Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:47.8180308Z 2025-10-10T01:58:47.8187873Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:47.8188441Z 2025-10-10T01:58:47.8188771Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] raise RuntimeError( 2025-10-10T01:58:47.8189062Z 2025-10-10T01:58:47.8189662Z (EngineCore_DP0 pid=10978) ERROR 10-10 01:58:47 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:47.8190385Z (EngineCore_DP0 pid=10978) Process EngineCore_DP0: 2025-10-10T01:58:47.8190833Z (EngineCore_DP0 pid=10978) Traceback (most recent call last): 2025-10-10T01:58:47.8191467Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:47.8191991Z (EngineCore_DP0 pid=10978) self.run() 2025-10-10T01:58:47.8192541Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:47.8193376Z (EngineCore_DP0 pid=10978) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:47.8194115Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:47.8194679Z (EngineCore_DP0 pid=10978) raise e 2025-10-10T01:58:47.8195277Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:47.8195917Z (EngineCore_DP0 pid=10978) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:47.8196592Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8197244Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:47.8198036Z (EngineCore_DP0 pid=10978) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:47.8198786Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:47.8199600Z (EngineCore_DP0 pid=10978) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:47.8200075Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8200746Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:47.8201341Z (EngineCore_DP0 pid=10978) self._init_executor() 2025-10-10T01:58:47.8202026Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:47.8202738Z (EngineCore_DP0 pid=10978) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:47.8203475Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:47.8204541Z (EngineCore_DP0 pid=10978) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:47.8205273Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8205960Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:47.8206574Z (EngineCore_DP0 pid=10978) return func(*args, **kwargs) 2025-10-10T01:58:47.8206984Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8207649Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:47.8208299Z (EngineCore_DP0 pid=10978) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:47.8208736Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8209416Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:47.8210075Z (EngineCore_DP0 pid=10978) module = importlib.import_module(module_name) 2025-10-10T01:58:47.8210531Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8211117Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:47.8211767Z (EngineCore_DP0 pid=10978) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:47.8212268Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8212768Z (EngineCore_DP0 pid=10978) File "", line 1387, in _gcd_import 2025-10-10T01:58:47.8213330Z (EngineCore_DP0 pid=10978) File "", line 1360, in _find_and_load 2025-10-10T01:58:47.8213925Z (EngineCore_DP0 pid=10978) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:47.8214528Z (EngineCore_DP0 pid=10978) File "", line 935, in _load_unlocked 2025-10-10T01:58:47.8215106Z (EngineCore_DP0 pid=10978) File "", line 999, in exec_module 2025-10-10T01:58:47.8216126Z (EngineCore_DP0 pid=10978) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:47.8216957Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:47.8217692Z (EngineCore_DP0 pid=10978) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:47.8218448Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:47.8219196Z (EngineCore_DP0 pid=10978) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:47.8220002Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:47.8220686Z (EngineCore_DP0 pid=10978) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:47.8221483Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:47.8222310Z (EngineCore_DP0 pid=10978) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:47.8222863Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8223586Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:47.8224272Z (EngineCore_DP0 pid=10978) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:47.8224718Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8225476Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:47.8226205Z (EngineCore_DP0 pid=10978) return _is_fa2_supported(device)[0] 2025-10-10T01:58:47.8226870Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8227634Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:47.8228369Z (EngineCore_DP0 pid=10978) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:47.8228836Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8229517Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:47.8230163Z (EngineCore_DP0 pid=10978) prop = get_device_properties(device) 2025-10-10T01:58:47.8230595Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:47.8231264Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:47.8231935Z (EngineCore_DP0 pid=10978) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:47.8232339Z (EngineCore_DP0 pid=10978) ^^^^^^^^^^^^ 2025-10-10T01:58:47.8232935Z (EngineCore_DP0 pid=10978) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:47.8233520Z (EngineCore_DP0 pid=10978) raise RuntimeError( 2025-10-10T01:58:47.8234213Z (EngineCore_DP0 pid=10978) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:48.2285077Z FAILED 2025-10-10T01:58:48.2413942Z models/test_initialization.py::test_can_initialize_large_subset[Cohere2ForCausalLM] Fork a new process to run a test 10982 2025-10-10T01:58:48.2425468Z Fork a new process to run a test 0 2025-10-10T01:58:48.2697029Z INFO 10-10 01:58:48 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Cohere2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'CohereForAI/c4ai-command-r7b-12-2024'} 2025-10-10T01:58:48.5287047Z 2025-10-10T01:58:48.5288279Z config.json: 0% 0.00/1.02k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:50.9338711Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:50.9339410Z 2025-10-10T01:58:50.9339952Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:50.9340369Z 2025-10-10T01:58:50.9341052Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:50.9341658Z 2025-10-10T01:58:50.9342053Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:50.9342442Z 2025-10-10T01:58:50.9343054Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9343420Z 2025-10-10T01:58:50.9344025Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:50.9344579Z 2025-10-10T01:58:50.9345056Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:50.9345746Z 2025-10-10T01:58:50.9346408Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:50.9347002Z 2025-10-10T01:58:50.9347413Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:50.9347812Z 2025-10-10T01:58:50.9348171Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9348517Z 2025-10-10T01:58:50.9349179Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:50.9349937Z 2025-10-10T01:58:50.9350411Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] self._init_executor() 2025-10-10T01:58:50.9350913Z 2025-10-10T01:58:50.9351962Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:50.9352846Z 2025-10-10T01:58:50.9353438Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:50.9353802Z 2025-10-10T01:58:50.9354368Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:50.9354889Z 2025-10-10T01:58:50.9355236Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:50.9355584Z 2025-10-10T01:58:50.9355874Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9356318Z 2025-10-10T01:58:50.9356845Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:50.9357400Z 2025-10-10T01:58:50.9357776Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:50.9358058Z 2025-10-10T01:58:50.9358329Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9358589Z 2025-10-10T01:58:50.9359243Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:50.9359733Z 2025-10-10T01:58:50.9360028Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:50.9360327Z 2025-10-10T01:58:50.9360598Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9360877Z 2025-10-10T01:58:50.9361409Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:50.9361904Z 2025-10-10T01:58:50.9362269Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:50.9362574Z 2025-10-10T01:58:50.9362858Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9363134Z 2025-10-10T01:58:50.9363578Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:50.9364003Z 2025-10-10T01:58:50.9364350Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:50.9364679Z 2025-10-10T01:58:50.9364976Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9365254Z 2025-10-10T01:58:50.9365596Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:50.9365929Z 2025-10-10T01:58:50.9366278Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:50.9366623Z 2025-10-10T01:58:50.9366999Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:50.9367377Z 2025-10-10T01:58:50.9367725Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:50.9368057Z 2025-10-10T01:58:50.9368425Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:50.9368776Z 2025-10-10T01:58:50.9369158Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:50.9369524Z 2025-10-10T01:58:50.9370031Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:50.9370491Z 2025-10-10T01:58:50.9370845Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:50.9371892Z 2025-10-10T01:58:50.9372479Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:50.9373019Z 2025-10-10T01:58:50.9373398Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:50.9373791Z 2025-10-10T01:58:50.9374359Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:50.9374881Z 2025-10-10T01:58:50.9375177Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:50.9375488Z 2025-10-10T01:58:50.9376116Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:50.9376688Z 2025-10-10T01:58:50.9377065Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:50.9377433Z 2025-10-10T01:58:50.9377741Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9378011Z 2025-10-10T01:58:50.9378584Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:50.9379104Z 2025-10-10T01:58:50.9379415Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:50.9379724Z 2025-10-10T01:58:50.9379997Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9380276Z 2025-10-10T01:58:50.9380882Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:50.9381447Z 2025-10-10T01:58:50.9381728Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:50.9382023Z 2025-10-10T01:58:50.9382290Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9382558Z 2025-10-10T01:58:50.9383137Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:50.9383663Z 2025-10-10T01:58:50.9383991Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:50.9384303Z 2025-10-10T01:58:50.9384585Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9384859Z 2025-10-10T01:58:50.9385389Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:50.9385871Z 2025-10-10T01:58:50.9386151Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:50.9386442Z 2025-10-10T01:58:50.9386709Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9387047Z 2025-10-10T01:58:50.9387573Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:50.9388112Z 2025-10-10T01:58:50.9388480Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:50.9388793Z 2025-10-10T01:58:50.9389028Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:50.9389272Z 2025-10-10T01:58:50.9389767Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:50.9390225Z 2025-10-10T01:58:50.9390483Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] raise RuntimeError( 2025-10-10T01:58:50.9390757Z 2025-10-10T01:58:50.9391324Z (EngineCore_DP0 pid=11010) ERROR 10-10 01:58:50 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:50.9392019Z (EngineCore_DP0 pid=11010) Process EngineCore_DP0: 2025-10-10T01:58:50.9392419Z (EngineCore_DP0 pid=11010) Traceback (most recent call last): 2025-10-10T01:58:50.9393082Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:50.9393611Z (EngineCore_DP0 pid=11010) self.run() 2025-10-10T01:58:50.9394135Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:50.9394701Z (EngineCore_DP0 pid=11010) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:50.9395386Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:50.9395937Z (EngineCore_DP0 pid=11010) raise e 2025-10-10T01:58:50.9396988Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:50.9397665Z (EngineCore_DP0 pid=11010) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:50.9398107Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9398744Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:50.9399462Z (EngineCore_DP0 pid=11010) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:50.9400149Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:50.9400774Z (EngineCore_DP0 pid=11010) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:50.9401242Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9401903Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:50.9402492Z (EngineCore_DP0 pid=11010) self._init_executor() 2025-10-10T01:58:50.9403162Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:50.9403851Z (EngineCore_DP0 pid=11010) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:50.9404733Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:50.9405454Z (EngineCore_DP0 pid=11010) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:50.9406014Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9406742Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:50.9407346Z (EngineCore_DP0 pid=11010) return func(*args, **kwargs) 2025-10-10T01:58:50.9407737Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9408385Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:50.9409031Z (EngineCore_DP0 pid=11010) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:50.9409470Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9410154Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:50.9410825Z (EngineCore_DP0 pid=11010) module = importlib.import_module(module_name) 2025-10-10T01:58:50.9411342Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9411951Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:50.9412585Z (EngineCore_DP0 pid=11010) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:50.9413080Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9413580Z (EngineCore_DP0 pid=11010) File "", line 1387, in _gcd_import 2025-10-10T01:58:50.9414137Z (EngineCore_DP0 pid=11010) File "", line 1360, in _find_and_load 2025-10-10T01:58:50.9414726Z (EngineCore_DP0 pid=11010) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:50.9415313Z (EngineCore_DP0 pid=11010) File "", line 935, in _load_unlocked 2025-10-10T01:58:50.9415888Z (EngineCore_DP0 pid=11010) File "", line 999, in exec_module 2025-10-10T01:58:50.9416491Z (EngineCore_DP0 pid=11010) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:50.9417233Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:50.9417915Z (EngineCore_DP0 pid=11010) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:50.9418643Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:50.9419370Z (EngineCore_DP0 pid=11010) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:50.9420156Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:50.9420854Z (EngineCore_DP0 pid=11010) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:50.9421648Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:50.9422520Z (EngineCore_DP0 pid=11010) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:50.9423005Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9423754Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:50.9424495Z (EngineCore_DP0 pid=11010) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:50.9424941Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9425681Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:50.9426384Z (EngineCore_DP0 pid=11010) return _is_fa2_supported(device)[0] 2025-10-10T01:58:50.9426806Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9427521Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:50.9428253Z (EngineCore_DP0 pid=11010) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:50.9428760Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9429441Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:50.9430082Z (EngineCore_DP0 pid=11010) prop = get_device_properties(device) 2025-10-10T01:58:50.9430504Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:50.9431166Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:50.9431829Z (EngineCore_DP0 pid=11010) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:50.9432245Z (EngineCore_DP0 pid=11010) ^^^^^^^^^^^^ 2025-10-10T01:58:50.9432836Z (EngineCore_DP0 pid=11010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:50.9433406Z (EngineCore_DP0 pid=11010) raise RuntimeError( 2025-10-10T01:58:50.9434094Z (EngineCore_DP0 pid=11010) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:51.3609724Z FAILED 2025-10-10T01:58:51.3739298Z models/test_initialization.py::test_can_initialize_large_subset[EagleLlama4ForCausalLM] Fork a new process to run a test 11014 2025-10-10T01:58:51.3750368Z Fork a new process to run a test 0 2025-10-10T01:58:51.4029605Z INFO 10-10 01:58:51 [utils.py:233] non-default args: {'tokenizer': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='EagleLlama4ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'speculative_config': {'model': 'morgendave/EAGLE-Llama-4-Scout-17B-16E-Instruct', 'num_speculative_tokens': 1}, 'model_impl': 'vllm', 'model': 'morgendave/EAGLE-Llama-4-Scout-17B-16E-Instruct'} 2025-10-10T01:58:51.5783622Z 2025-10-10T01:58:51.5785773Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:58:51.5786088Z config.json: 1.51kB [00:00, 7.67MB/s] 2025-10-10T01:58:51.7225320Z 2025-10-10T01:58:51.7226273Z preprocessor_config.json: 0% 0.00/636 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:58:59.8557715Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] EngineCore failed to start. 2025-10-10T01:58:59.8558115Z 2025-10-10T01:58:59.8558590Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] Traceback (most recent call last): 2025-10-10T01:58:59.8558889Z 2025-10-10T01:58:59.8559574Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:59.8560056Z 2025-10-10T01:58:59.8560371Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:59.8561654Z 2025-10-10T01:58:59.8561952Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8562252Z 2025-10-10T01:58:59.8562857Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:59.8563404Z 2025-10-10T01:58:59.8563877Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:59.8564253Z 2025-10-10T01:58:59.8564748Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:59.8565191Z 2025-10-10T01:58:59.8565510Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:59.8565825Z 2025-10-10T01:58:59.8566110Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8566386Z 2025-10-10T01:58:59.8567018Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:59.8567533Z 2025-10-10T01:58:59.8567808Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] self._init_executor() 2025-10-10T01:58:59.8568322Z 2025-10-10T01:58:59.8569055Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:59.8569716Z 2025-10-10T01:58:59.8570236Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:59.8570775Z 2025-10-10T01:58:59.8571470Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:59.8572082Z 2025-10-10T01:58:59.8572589Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:59.8572940Z 2025-10-10T01:58:59.8573246Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8573531Z 2025-10-10T01:58:59.8574030Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:59.8574488Z 2025-10-10T01:58:59.8574763Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:58:59.8575032Z 2025-10-10T01:58:59.8575290Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8575554Z 2025-10-10T01:58:59.8576071Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:59.8576560Z 2025-10-10T01:58:59.8576848Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:59.8577152Z 2025-10-10T01:58:59.8577416Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8577694Z 2025-10-10T01:58:59.8578298Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:59.8578781Z 2025-10-10T01:58:59.8579130Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:58:59.8579483Z 2025-10-10T01:58:59.8579772Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8580052Z 2025-10-10T01:58:59.8580498Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:59.8580908Z 2025-10-10T01:58:59.8581300Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:59.8581634Z 2025-10-10T01:58:59.8581918Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8582194Z 2025-10-10T01:58:59.8582541Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:58:59.8582870Z 2025-10-10T01:58:59.8583277Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:58:59.8583624Z 2025-10-10T01:58:59.8584007Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:59.8584362Z 2025-10-10T01:58:59.8584706Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:58:59.8585061Z 2025-10-10T01:58:59.8585432Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:58:59.8585805Z 2025-10-10T01:58:59.8586182Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:59.8586554Z 2025-10-10T01:58:59.8587085Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:59.8587574Z 2025-10-10T01:58:59.8587935Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:59.8588267Z 2025-10-10T01:58:59.8588791Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:59.8589271Z 2025-10-10T01:58:59.8589649Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:59.8590003Z 2025-10-10T01:58:59.8590561Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:59.8591082Z 2025-10-10T01:58:59.8591384Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:58:59.8591680Z 2025-10-10T01:58:59.8592307Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:59.8592936Z 2025-10-10T01:58:59.8593302Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:59.8593658Z 2025-10-10T01:58:59.8593952Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8594254Z 2025-10-10T01:58:59.8594815Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:59.8595346Z 2025-10-10T01:58:59.8595645Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:58:59.8595939Z 2025-10-10T01:58:59.8596559Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8596850Z 2025-10-10T01:58:59.8597469Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:59.8598022Z 2025-10-10T01:58:59.8598311Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:58:59.8598601Z 2025-10-10T01:58:59.8598963Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8599359Z 2025-10-10T01:58:59.8599973Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:59.8600527Z 2025-10-10T01:58:59.8600850Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:59.8601171Z 2025-10-10T01:58:59.8601446Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8601727Z 2025-10-10T01:58:59.8602256Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:59.8602747Z 2025-10-10T01:58:59.8603038Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:58:59.8603323Z 2025-10-10T01:58:59.8603596Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8603868Z 2025-10-10T01:58:59.8604407Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:59.8604897Z 2025-10-10T01:58:59.8605210Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:58:59.8605524Z 2025-10-10T01:58:59.8605753Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:58:59.8606005Z 2025-10-10T01:58:59.8606489Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:59.8606945Z 2025-10-10T01:58:59.8607190Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] raise RuntimeError( 2025-10-10T01:58:59.8607450Z 2025-10-10T01:58:59.8608033Z (EngineCore_DP0 pid=11072) ERROR 10-10 01:58:59 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:58:59.8608850Z (EngineCore_DP0 pid=11072) Process EngineCore_DP0: 2025-10-10T01:58:59.8609253Z (EngineCore_DP0 pid=11072) Traceback (most recent call last): 2025-10-10T01:58:59.8609945Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:58:59.8610562Z (EngineCore_DP0 pid=11072) self.run() 2025-10-10T01:58:59.8611119Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:58:59.8611694Z (EngineCore_DP0 pid=11072) self._target(*self._args, **self._kwargs) 2025-10-10T01:58:59.8612487Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:58:59.8613401Z (EngineCore_DP0 pid=11072) raise e 2025-10-10T01:58:59.8614395Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:58:59.8615454Z (EngineCore_DP0 pid=11072) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:58:59.8616306Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8617620Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:58:59.8618809Z (EngineCore_DP0 pid=11072) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:58:59.8619890Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:58:59.8620582Z (EngineCore_DP0 pid=11072) self.model_executor = executor_class(vllm_config) 2025-10-10T01:58:59.8621066Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8621753Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:58:59.8622351Z (EngineCore_DP0 pid=11072) self._init_executor() 2025-10-10T01:58:59.8623029Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:58:59.8623731Z (EngineCore_DP0 pid=11072) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:58:59.8624475Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:58:59.8625200Z (EngineCore_DP0 pid=11072) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:58:59.8625706Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8626357Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:58:59.8626951Z (EngineCore_DP0 pid=11072) return func(*args, **kwargs) 2025-10-10T01:58:59.8627342Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8627998Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:58:59.8628648Z (EngineCore_DP0 pid=11072) worker_class = resolve_obj_by_qualname( 2025-10-10T01:58:59.8629086Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8629764Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:58:59.8630501Z (EngineCore_DP0 pid=11072) module = importlib.import_module(module_name) 2025-10-10T01:58:59.8630996Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8631628Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:58:59.8632262Z (EngineCore_DP0 pid=11072) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:58:59.8632769Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8633257Z (EngineCore_DP0 pid=11072) File "", line 1387, in _gcd_import 2025-10-10T01:58:59.8633811Z (EngineCore_DP0 pid=11072) File "", line 1360, in _find_and_load 2025-10-10T01:58:59.8634404Z (EngineCore_DP0 pid=11072) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:58:59.8634985Z (EngineCore_DP0 pid=11072) File "", line 935, in _load_unlocked 2025-10-10T01:58:59.8635558Z (EngineCore_DP0 pid=11072) File "", line 999, in exec_module 2025-10-10T01:58:59.8636206Z (EngineCore_DP0 pid=11072) File "", line 488, in _call_with_frames_removed 2025-10-10T01:58:59.8636962Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:58:59.8637650Z (EngineCore_DP0 pid=11072) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:58:59.8638377Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:58:59.8639205Z (EngineCore_DP0 pid=11072) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:58:59.8640005Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:58:59.8640674Z (EngineCore_DP0 pid=11072) class FlashAttentionMetadataBuilder( 2025-10-10T01:58:59.8641467Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:58:59.8642301Z (EngineCore_DP0 pid=11072) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:58:59.8642787Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8643488Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:58:59.8644172Z (EngineCore_DP0 pid=11072) if not is_fa_version_supported(fa_version): 2025-10-10T01:58:59.8644617Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8645376Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:58:59.8646084Z (EngineCore_DP0 pid=11072) return _is_fa2_supported(device)[0] 2025-10-10T01:58:59.8646503Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8647213Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:58:59.8648004Z (EngineCore_DP0 pid=11072) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:58:59.8648469Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8649188Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:58:59.8649877Z (EngineCore_DP0 pid=11072) prop = get_device_properties(device) 2025-10-10T01:58:59.8650306Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:58:59.8651004Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:58:59.8651681Z (EngineCore_DP0 pid=11072) _lazy_init() # will define _get_device_properties 2025-10-10T01:58:59.8652087Z (EngineCore_DP0 pid=11072) ^^^^^^^^^^^^ 2025-10-10T01:58:59.8652691Z (EngineCore_DP0 pid=11072) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:58:59.8653266Z (EngineCore_DP0 pid=11072) raise RuntimeError( 2025-10-10T01:58:59.8653956Z (EngineCore_DP0 pid=11072) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:00.2821166Z FAILED 2025-10-10T01:59:00.2951432Z models/test_initialization.py::test_can_initialize_large_subset[NemotronForCausalLM] Fork a new process to run a test 11076 2025-10-10T01:59:00.2963559Z Fork a new process to run a test 0 2025-10-10T01:59:00.3238430Z INFO 10-10 01:59:00 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='NemotronForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'nvidia/Minitron-8B-Base'} 2025-10-10T01:59:00.4061565Z 2025-10-10T01:59:00.4063138Z config.json: 0% 0.00/635 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:59:09.2451804Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] EngineCore failed to start. 2025-10-10T01:59:09.2452468Z 2025-10-10T01:59:09.2453081Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] Traceback (most recent call last): 2025-10-10T01:59:09.2453492Z 2025-10-10T01:59:09.2454212Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:09.2454830Z 2025-10-10T01:59:09.2455229Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:09.2455612Z 2025-10-10T01:59:09.2455962Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2456327Z 2025-10-10T01:59:09.2456852Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:09.2457306Z 2025-10-10T01:59:09.2457662Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:09.2457993Z 2025-10-10T01:59:09.2458648Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:09.2459112Z 2025-10-10T01:59:09.2459444Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:09.2459761Z 2025-10-10T01:59:09.2460036Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2460317Z 2025-10-10T01:59:09.2460826Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:09.2461561Z 2025-10-10T01:59:09.2461843Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] self._init_executor() 2025-10-10T01:59:09.2462345Z 2025-10-10T01:59:09.2463095Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:09.2464208Z 2025-10-10T01:59:09.2464884Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:09.2465502Z 2025-10-10T01:59:09.2466537Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:09.2467087Z 2025-10-10T01:59:09.2467481Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:09.2467820Z 2025-10-10T01:59:09.2468115Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2468406Z 2025-10-10T01:59:09.2469034Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:09.2469506Z 2025-10-10T01:59:09.2469778Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:59:09.2470053Z 2025-10-10T01:59:09.2470302Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2470564Z 2025-10-10T01:59:09.2471079Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:09.2471557Z 2025-10-10T01:59:09.2471856Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:09.2472164Z 2025-10-10T01:59:09.2472449Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2472733Z 2025-10-10T01:59:09.2473284Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:09.2473774Z 2025-10-10T01:59:09.2474076Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:59:09.2474384Z 2025-10-10T01:59:09.2474709Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2474989Z 2025-10-10T01:59:09.2475440Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:09.2475856Z 2025-10-10T01:59:09.2476212Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:09.2476540Z 2025-10-10T01:59:09.2476827Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2477110Z 2025-10-10T01:59:09.2477451Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:59:09.2477799Z 2025-10-10T01:59:09.2478149Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:59:09.2478558Z 2025-10-10T01:59:09.2478984Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:09.2479483Z 2025-10-10T01:59:09.2479885Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:59:09.2480231Z 2025-10-10T01:59:09.2480599Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:59:09.2480951Z 2025-10-10T01:59:09.2481329Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:09.2481686Z 2025-10-10T01:59:09.2482187Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:09.2482654Z 2025-10-10T01:59:09.2483017Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:09.2483358Z 2025-10-10T01:59:09.2483929Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:09.2484427Z 2025-10-10T01:59:09.2484803Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:09.2485167Z 2025-10-10T01:59:09.2485715Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:09.2486247Z 2025-10-10T01:59:09.2486543Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:59:09.2486847Z 2025-10-10T01:59:09.2487476Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:09.2488054Z 2025-10-10T01:59:09.2488428Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:09.2488779Z 2025-10-10T01:59:09.2489043Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2489305Z 2025-10-10T01:59:09.2489874Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:09.2490391Z 2025-10-10T01:59:09.2490695Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:59:09.2490989Z 2025-10-10T01:59:09.2491263Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2491544Z 2025-10-10T01:59:09.2492142Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:09.2492707Z 2025-10-10T01:59:09.2492993Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:59:09.2493281Z 2025-10-10T01:59:09.2493607Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2493878Z 2025-10-10T01:59:09.2494519Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:09.2495105Z 2025-10-10T01:59:09.2495443Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:09.2495758Z 2025-10-10T01:59:09.2496046Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2496570Z 2025-10-10T01:59:09.2497111Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:09.2497601Z 2025-10-10T01:59:09.2497893Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:59:09.2498185Z 2025-10-10T01:59:09.2498454Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2498738Z 2025-10-10T01:59:09.2499344Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:09.2499850Z 2025-10-10T01:59:09.2500165Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:59:09.2500469Z 2025-10-10T01:59:09.2500700Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:59:09.2500941Z 2025-10-10T01:59:09.2501429Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:09.2501882Z 2025-10-10T01:59:09.2502133Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] raise RuntimeError( 2025-10-10T01:59:09.2502394Z 2025-10-10T01:59:09.2502960Z (EngineCore_DP0 pid=11155) ERROR 10-10 01:59:09 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:09.2503663Z (EngineCore_DP0 pid=11155) Process EngineCore_DP0: 2025-10-10T01:59:09.2504057Z (EngineCore_DP0 pid=11155) Traceback (most recent call last): 2025-10-10T01:59:09.2504664Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:59:09.2505180Z (EngineCore_DP0 pid=11155) self.run() 2025-10-10T01:59:09.2505726Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:59:09.2506301Z (EngineCore_DP0 pid=11155) self._target(*self._args, **self._kwargs) 2025-10-10T01:59:09.2506975Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:59:09.2507541Z (EngineCore_DP0 pid=11155) raise e 2025-10-10T01:59:09.2508130Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:09.2508763Z (EngineCore_DP0 pid=11155) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:09.2509201Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2509829Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:09.2510579Z (EngineCore_DP0 pid=11155) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:09.2511347Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:09.2512052Z (EngineCore_DP0 pid=11155) self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:09.2512503Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2513157Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:09.2513762Z (EngineCore_DP0 pid=11155) self._init_executor() 2025-10-10T01:59:09.2514427Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:09.2515119Z (EngineCore_DP0 pid=11155) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:09.2515846Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:09.2516621Z (EngineCore_DP0 pid=11155) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:09.2517128Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2517774Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:09.2518363Z (EngineCore_DP0 pid=11155) return func(*args, **kwargs) 2025-10-10T01:59:09.2518747Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2519463Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:09.2520115Z (EngineCore_DP0 pid=11155) worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:09.2520551Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2521222Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:09.2521881Z (EngineCore_DP0 pid=11155) module = importlib.import_module(module_name) 2025-10-10T01:59:09.2522324Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2522919Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:09.2523577Z (EngineCore_DP0 pid=11155) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:09.2524073Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2524573Z (EngineCore_DP0 pid=11155) File "", line 1387, in _gcd_import 2025-10-10T01:59:09.2525128Z (EngineCore_DP0 pid=11155) File "", line 1360, in _find_and_load 2025-10-10T01:59:09.2525713Z (EngineCore_DP0 pid=11155) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:09.2526305Z (EngineCore_DP0 pid=11155) File "", line 935, in _load_unlocked 2025-10-10T01:59:09.2526905Z (EngineCore_DP0 pid=11155) File "", line 999, in exec_module 2025-10-10T01:59:09.2527580Z (EngineCore_DP0 pid=11155) File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:09.2528318Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:09.2529056Z (EngineCore_DP0 pid=11155) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:09.2529849Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:09.2530576Z (EngineCore_DP0 pid=11155) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:09.2531362Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:09.2532039Z (EngineCore_DP0 pid=11155) class FlashAttentionMetadataBuilder( 2025-10-10T01:59:09.2532828Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:09.2533644Z (EngineCore_DP0 pid=11155) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:09.2534141Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2534881Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:09.2535574Z (EngineCore_DP0 pid=11155) if not is_fa_version_supported(fa_version): 2025-10-10T01:59:09.2536007Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2536749Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:09.2537467Z (EngineCore_DP0 pid=11155) return _is_fa2_supported(device)[0] 2025-10-10T01:59:09.2537881Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2538598Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:09.2539311Z (EngineCore_DP0 pid=11155) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:09.2539759Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2540426Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:09.2541076Z (EngineCore_DP0 pid=11155) prop = get_device_properties(device) 2025-10-10T01:59:09.2541496Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:09.2542156Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:09.2542814Z (EngineCore_DP0 pid=11155) _lazy_init() # will define _get_device_properties 2025-10-10T01:59:09.2543215Z (EngineCore_DP0 pid=11155) ^^^^^^^^^^^^ 2025-10-10T01:59:09.2543815Z (EngineCore_DP0 pid=11155) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:09.2544419Z (EngineCore_DP0 pid=11155) raise RuntimeError( 2025-10-10T01:59:09.2545116Z (EngineCore_DP0 pid=11155) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:09.6619628Z FAILED 2025-10-10T01:59:09.6749359Z models/test_initialization.py::test_can_initialize_large_subset[DbrxForCausalLM] Fork a new process to run a test 11159 2025-10-10T01:59:09.6759986Z Fork a new process to run a test 0 2025-10-10T01:59:09.7039664Z INFO 10-10 01:59:09 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DbrxForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'databricks/dbrx-instruct'} 2025-10-10T01:59:09.9546502Z 2025-10-10T01:59:09.9547399Z config.json: 0% 0.00/733 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:59:18.3067845Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] EngineCore failed to start. 2025-10-10T01:59:18.3068287Z 2025-10-10T01:59:18.3068887Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] Traceback (most recent call last): 2025-10-10T01:59:18.3069296Z 2025-10-10T01:59:18.3070042Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:18.3070937Z 2025-10-10T01:59:18.3071378Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:18.3071766Z 2025-10-10T01:59:18.3072119Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3072472Z 2025-10-10T01:59:18.3073083Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:18.3073674Z 2025-10-10T01:59:18.3074090Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:18.3074752Z 2025-10-10T01:59:18.3075304Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:18.3075761Z 2025-10-10T01:59:18.3076088Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:18.3076406Z 2025-10-10T01:59:18.3076690Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3076964Z 2025-10-10T01:59:18.3077475Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:18.3078074Z 2025-10-10T01:59:18.3078588Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] self._init_executor() 2025-10-10T01:59:18.3079230Z 2025-10-10T01:59:18.3080110Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:18.3080952Z 2025-10-10T01:59:18.3081368Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:18.3081860Z 2025-10-10T01:59:18.3082585Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:18.3083252Z 2025-10-10T01:59:18.3083627Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:18.3083978Z 2025-10-10T01:59:18.3084371Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3084741Z 2025-10-10T01:59:18.3085258Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:18.3085738Z 2025-10-10T01:59:18.3086019Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:59:18.3086296Z 2025-10-10T01:59:18.3086559Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3086824Z 2025-10-10T01:59:18.3087354Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:18.3087837Z 2025-10-10T01:59:18.3088129Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:18.3088436Z 2025-10-10T01:59:18.3088753Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3089044Z 2025-10-10T01:59:18.3089584Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:18.3090072Z 2025-10-10T01:59:18.3090373Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:59:18.3090680Z 2025-10-10T01:59:18.3090971Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3091248Z 2025-10-10T01:59:18.3091703Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:18.3092118Z 2025-10-10T01:59:18.3092464Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:18.3092794Z 2025-10-10T01:59:18.3093079Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3093361Z 2025-10-10T01:59:18.3093701Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:59:18.3094039Z 2025-10-10T01:59:18.3094402Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:59:18.3094741Z 2025-10-10T01:59:18.3095115Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:18.3095477Z 2025-10-10T01:59:18.3095819Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:59:18.3096450Z 2025-10-10T01:59:18.3096855Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:59:18.3097221Z 2025-10-10T01:59:18.3097606Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:18.3098060Z 2025-10-10T01:59:18.3098571Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:18.3099046Z 2025-10-10T01:59:18.3099465Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:18.3099876Z 2025-10-10T01:59:18.3100416Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:18.3100902Z 2025-10-10T01:59:18.3101275Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:18.3101641Z 2025-10-10T01:59:18.3102204Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:18.3102716Z 2025-10-10T01:59:18.3103010Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:59:18.3103305Z 2025-10-10T01:59:18.3104008Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:18.3104601Z 2025-10-10T01:59:18.3104983Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:18.3105337Z 2025-10-10T01:59:18.3105597Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3105857Z 2025-10-10T01:59:18.3106433Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:18.3106965Z 2025-10-10T01:59:18.3107262Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:59:18.3107562Z 2025-10-10T01:59:18.3107836Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3108116Z 2025-10-10T01:59:18.3108713Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:18.3109267Z 2025-10-10T01:59:18.3109547Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:59:18.3109839Z 2025-10-10T01:59:18.3110098Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3110373Z 2025-10-10T01:59:18.3110962Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:18.3111488Z 2025-10-10T01:59:18.3111812Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:18.3112121Z 2025-10-10T01:59:18.3112400Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3112672Z 2025-10-10T01:59:18.3113193Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:18.3113744Z 2025-10-10T01:59:18.3114029Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:59:18.3114329Z 2025-10-10T01:59:18.3114637Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3114974Z 2025-10-10T01:59:18.3115499Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:18.3115998Z 2025-10-10T01:59:18.3116318Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:59:18.3116624Z 2025-10-10T01:59:18.3116860Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:59:18.3117103Z 2025-10-10T01:59:18.3117594Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:18.3118051Z 2025-10-10T01:59:18.3118310Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] raise RuntimeError( 2025-10-10T01:59:18.3118571Z 2025-10-10T01:59:18.3119298Z (EngineCore_DP0 pid=11217) ERROR 10-10 01:59:18 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:18.3120052Z (EngineCore_DP0 pid=11217) Process EngineCore_DP0: 2025-10-10T01:59:18.3120447Z (EngineCore_DP0 pid=11217) Traceback (most recent call last): 2025-10-10T01:59:18.3121066Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:59:18.3121591Z (EngineCore_DP0 pid=11217) self.run() 2025-10-10T01:59:18.3122138Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:59:18.3122702Z (EngineCore_DP0 pid=11217) self._target(*self._args, **self._kwargs) 2025-10-10T01:59:18.3123389Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:59:18.3123948Z (EngineCore_DP0 pid=11217) raise e 2025-10-10T01:59:18.3124545Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:18.3125180Z (EngineCore_DP0 pid=11217) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:18.3125632Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3126254Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:18.3126901Z (EngineCore_DP0 pid=11217) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:18.3127561Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:18.3128198Z (EngineCore_DP0 pid=11217) self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:18.3128657Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3129305Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:18.3129895Z (EngineCore_DP0 pid=11217) self._init_executor() 2025-10-10T01:59:18.3130632Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:18.3131338Z (EngineCore_DP0 pid=11217) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:18.3132108Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:18.3132874Z (EngineCore_DP0 pid=11217) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:18.3133374Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3134017Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:18.3134614Z (EngineCore_DP0 pid=11217) return func(*args, **kwargs) 2025-10-10T01:59:18.3135002Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3135634Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:18.3136267Z (EngineCore_DP0 pid=11217) worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:18.3136697Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3137411Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:18.3138072Z (EngineCore_DP0 pid=11217) module = importlib.import_module(module_name) 2025-10-10T01:59:18.3138522Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3139112Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:18.3139744Z (EngineCore_DP0 pid=11217) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:18.3140237Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3140742Z (EngineCore_DP0 pid=11217) File "", line 1387, in _gcd_import 2025-10-10T01:59:18.3141306Z (EngineCore_DP0 pid=11217) File "", line 1360, in _find_and_load 2025-10-10T01:59:18.3141891Z (EngineCore_DP0 pid=11217) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:18.3142498Z (EngineCore_DP0 pid=11217) File "", line 935, in _load_unlocked 2025-10-10T01:59:18.3150006Z (EngineCore_DP0 pid=11217) File "", line 999, in exec_module 2025-10-10T01:59:18.3150702Z (EngineCore_DP0 pid=11217) File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:18.3151475Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:18.3152183Z (EngineCore_DP0 pid=11217) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:18.3152933Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:18.3153665Z (EngineCore_DP0 pid=11217) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:18.3154468Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:18.3155267Z (EngineCore_DP0 pid=11217) class FlashAttentionMetadataBuilder( 2025-10-10T01:59:18.3156069Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:18.3156932Z (EngineCore_DP0 pid=11217) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:18.3157479Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3158188Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:18.3158890Z (EngineCore_DP0 pid=11217) if not is_fa_version_supported(fa_version): 2025-10-10T01:59:18.3159426Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3160183Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:18.3160893Z (EngineCore_DP0 pid=11217) return _is_fa2_supported(device)[0] 2025-10-10T01:59:18.3161310Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3162088Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:18.3162820Z (EngineCore_DP0 pid=11217) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:18.3163289Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3163964Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:18.3164598Z (EngineCore_DP0 pid=11217) prop = get_device_properties(device) 2025-10-10T01:59:18.3165025Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:18.3165696Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:18.3166370Z (EngineCore_DP0 pid=11217) _lazy_init() # will define _get_device_properties 2025-10-10T01:59:18.3166772Z (EngineCore_DP0 pid=11217) ^^^^^^^^^^^^ 2025-10-10T01:59:18.3167361Z (EngineCore_DP0 pid=11217) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:18.3167937Z (EngineCore_DP0 pid=11217) raise RuntimeError( 2025-10-10T01:59:18.3168635Z (EngineCore_DP0 pid=11217) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:18.7180505Z FAILED 2025-10-10T01:59:18.7308568Z models/test_initialization.py::test_can_initialize_large_subset[JambaForSequenceClassification] Fork a new process to run a test 11221 2025-10-10T01:59:18.7320976Z Fork a new process to run a test 0 2025-10-10T01:59:18.7597779Z INFO 10-10 01:59:18 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='JambaForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ai21labs/Jamba-tiny-reward-dev'} 2025-10-10T01:59:19.1526768Z 2025-10-10T01:59:19.1528655Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:59:19.1528960Z config.json: 1.14kB [00:00, 6.34MB/s] 2025-10-10T01:59:25.8853206Z INFO 10-10 01:59:25 [model.py:809] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message. 2025-10-10T01:59:25.8854265Z INFO 10-10 01:59:25 [model.py:551] Resolved architecture: JambaForSequenceClassification 2025-10-10T01:59:25.8854741Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T01:59:25.9105959Z INFO 10-10 01:59:25 [model.py:1545] Using max model len 262144 2025-10-10T01:59:25.9416418Z INFO 10-10 01:59:25 [arg_utils.py:1580] (Enabling) chunked prefill by default 2025-10-10T01:59:25.9417022Z INFO 10-10 01:59:25 [arg_utils.py:1583] (Enabling) prefix caching by default 2025-10-10T01:59:26.0835979Z INFO 10-10 01:59:26 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T01:59:26.0837191Z INFO 10-10 01:59:26 [config.py:297] Hybrid or mamba-based model detected: disabling prefix caching since it is not yet supported. 2025-10-10T01:59:26.0838223Z INFO 10-10 01:59:26 [config.py:308] Hybrid or mamba-based model detected: setting cudagraph mode to FULL_AND_PIECEWISE in order to optimize performance. 2025-10-10T01:59:26.1345139Z INFO 10-10 01:59:26 [config.py:376] Setting attention block size to 80 tokens to ensure that attention page size is >= mamba page size. 2025-10-10T01:59:26.1348077Z INFO 10-10 01:59:26 [config.py:397] Padding mamba page size by 5.26% to ensure that mamba page size and attention page size are exactly equal. 2025-10-10T01:59:26.1813507Z 2025-10-10T01:59:26.1815889Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:59:26.1816570Z tokenizer_config.json: 14.3kB [00:00, 68.9MB/s] 2025-10-10T01:59:26.3223587Z 2025-10-10T01:59:26.6022705Z tokenizer.model: 0% 0.00/1.12M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:59:27.1193945Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] EngineCore failed to start. 2025-10-10T01:59:27.1194561Z 2025-10-10T01:59:27.1194975Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] Traceback (most recent call last): 2025-10-10T01:59:27.1195352Z 2025-10-10T01:59:27.1196037Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:27.1197285Z 2025-10-10T01:59:27.1198089Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:27.1198522Z 2025-10-10T01:59:27.1198900Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1199390Z 2025-10-10T01:59:27.1200019Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:27.1200601Z 2025-10-10T01:59:27.1201035Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:27.1201436Z 2025-10-10T01:59:27.1202063Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:27.1202631Z 2025-10-10T01:59:27.1203044Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:27.1203351Z 2025-10-10T01:59:27.1203651Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1203930Z 2025-10-10T01:59:27.1204466Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:27.1204938Z 2025-10-10T01:59:27.1205206Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] self._init_executor() 2025-10-10T01:59:27.1205478Z 2025-10-10T01:59:27.1206030Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:27.1206561Z 2025-10-10T01:59:27.1206892Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:27.1207219Z 2025-10-10T01:59:27.1207776Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:27.1208284Z 2025-10-10T01:59:27.1208646Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:27.1209118Z 2025-10-10T01:59:27.1209426Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1209721Z 2025-10-10T01:59:27.1210319Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:27.1210855Z 2025-10-10T01:59:27.1211152Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:59:27.1211468Z 2025-10-10T01:59:27.1211729Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1211987Z 2025-10-10T01:59:27.1212524Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:27.1213016Z 2025-10-10T01:59:27.1213325Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:27.1213616Z 2025-10-10T01:59:27.1213901Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1214176Z 2025-10-10T01:59:27.1214768Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:27.1215266Z 2025-10-10T01:59:27.1215587Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:59:27.1215898Z 2025-10-10T01:59:27.1216187Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1216472Z 2025-10-10T01:59:27.1216930Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:27.1217368Z 2025-10-10T01:59:27.1217726Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:27.1218060Z 2025-10-10T01:59:27.1218364Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1218641Z 2025-10-10T01:59:27.1218997Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:59:27.1219328Z 2025-10-10T01:59:27.1219691Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:59:27.1220035Z 2025-10-10T01:59:27.1220434Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:27.1220795Z 2025-10-10T01:59:27.1221157Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:59:27.1221491Z 2025-10-10T01:59:27.1221871Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:59:27.1222223Z 2025-10-10T01:59:27.1222612Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:27.1222974Z 2025-10-10T01:59:27.1223480Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:27.1224030Z 2025-10-10T01:59:27.1224399Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:27.1224788Z 2025-10-10T01:59:27.1225385Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:27.1225874Z 2025-10-10T01:59:27.1226262Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:27.1226630Z 2025-10-10T01:59:27.1227193Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:27.1227705Z 2025-10-10T01:59:27.1228019Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:59:27.1228318Z 2025-10-10T01:59:27.1228960Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:27.1229537Z 2025-10-10T01:59:27.1229966Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:27.1230322Z 2025-10-10T01:59:27.1230595Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1230859Z 2025-10-10T01:59:27.1231431Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:27.1231974Z 2025-10-10T01:59:27.1232286Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:59:27.1232593Z 2025-10-10T01:59:27.1232878Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1233158Z 2025-10-10T01:59:27.1233770Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:27.1234325Z 2025-10-10T01:59:27.1234615Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:59:27.1234899Z 2025-10-10T01:59:27.1235177Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1235449Z 2025-10-10T01:59:27.1236041Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:27.1236570Z 2025-10-10T01:59:27.1236909Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:27.1237221Z 2025-10-10T01:59:27.1237503Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1237781Z 2025-10-10T01:59:27.1238314Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:27.1238805Z 2025-10-10T01:59:27.1239196Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:59:27.1239554Z 2025-10-10T01:59:27.1239843Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1240116Z 2025-10-10T01:59:27.1240712Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:27.1241245Z 2025-10-10T01:59:27.1241582Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:59:27.1241893Z 2025-10-10T01:59:27.1242140Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:59:27.1242384Z 2025-10-10T01:59:27.1242892Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:27.1243349Z 2025-10-10T01:59:27.1243611Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] raise RuntimeError( 2025-10-10T01:59:27.1243876Z 2025-10-10T01:59:27.1244458Z (EngineCore_DP0 pid=11300) ERROR 10-10 01:59:27 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:27.1245211Z (EngineCore_DP0 pid=11300) Process EngineCore_DP0: 2025-10-10T01:59:27.1245629Z (EngineCore_DP0 pid=11300) Traceback (most recent call last): 2025-10-10T01:59:27.1246250Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:59:27.1246768Z (EngineCore_DP0 pid=11300) self.run() 2025-10-10T01:59:27.1247311Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:59:27.1247890Z (EngineCore_DP0 pid=11300) self._target(*self._args, **self._kwargs) 2025-10-10T01:59:27.1248574Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:59:27.1249150Z (EngineCore_DP0 pid=11300) raise e 2025-10-10T01:59:27.1249750Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:27.1250401Z (EngineCore_DP0 pid=11300) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:27.1250862Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1251500Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:27.1252162Z (EngineCore_DP0 pid=11300) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:27.1252843Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:27.1253478Z (EngineCore_DP0 pid=11300) self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:27.1253953Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1254622Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:27.1255224Z (EngineCore_DP0 pid=11300) self._init_executor() 2025-10-10T01:59:27.1255904Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:27.1256663Z (EngineCore_DP0 pid=11300) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:27.1257467Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:27.1258241Z (EngineCore_DP0 pid=11300) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:27.1258756Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1259410Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:27.1260010Z (EngineCore_DP0 pid=11300) return func(*args, **kwargs) 2025-10-10T01:59:27.1260405Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1261060Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:27.1261703Z (EngineCore_DP0 pid=11300) worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:27.1262144Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1262858Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:27.1263539Z (EngineCore_DP0 pid=11300) module = importlib.import_module(module_name) 2025-10-10T01:59:27.1263995Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1264589Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:27.1265221Z (EngineCore_DP0 pid=11300) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:27.1265715Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1266228Z (EngineCore_DP0 pid=11300) File "", line 1387, in _gcd_import 2025-10-10T01:59:27.1266797Z (EngineCore_DP0 pid=11300) File "", line 1360, in _find_and_load 2025-10-10T01:59:27.1267399Z (EngineCore_DP0 pid=11300) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:27.1267987Z (EngineCore_DP0 pid=11300) File "", line 935, in _load_unlocked 2025-10-10T01:59:27.1268570Z (EngineCore_DP0 pid=11300) File "", line 999, in exec_module 2025-10-10T01:59:27.1269176Z (EngineCore_DP0 pid=11300) File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:27.1269921Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:27.1270613Z (EngineCore_DP0 pid=11300) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:27.1271354Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:27.1272092Z (EngineCore_DP0 pid=11300) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:27.1272877Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:27.1273565Z (EngineCore_DP0 pid=11300) class FlashAttentionMetadataBuilder( 2025-10-10T01:59:27.1274426Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:27.1275289Z (EngineCore_DP0 pid=11300) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:27.1275823Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1276534Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:27.1277228Z (EngineCore_DP0 pid=11300) if not is_fa_version_supported(fa_version): 2025-10-10T01:59:27.1277686Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1278447Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:27.1279249Z (EngineCore_DP0 pid=11300) return _is_fa2_supported(device)[0] 2025-10-10T01:59:27.1279686Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1280408Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:27.1281193Z (EngineCore_DP0 pid=11300) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:27.1281675Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1282359Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:27.1283026Z (EngineCore_DP0 pid=11300) prop = get_device_properties(device) 2025-10-10T01:59:27.1283456Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:27.1284126Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:27.1284800Z (EngineCore_DP0 pid=11300) _lazy_init() # will define _get_device_properties 2025-10-10T01:59:27.1285226Z (EngineCore_DP0 pid=11300) ^^^^^^^^^^^^ 2025-10-10T01:59:27.1285820Z (EngineCore_DP0 pid=11300) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:27.1286389Z (EngineCore_DP0 pid=11300) raise RuntimeError( 2025-10-10T01:59:27.1287096Z (EngineCore_DP0 pid=11300) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:27.5252587Z FAILED 2025-10-10T01:59:27.5382092Z models/test_initialization.py::test_can_initialize_large_subset[MantisForConditionalGeneration] Fork a new process to run a test 11304 2025-10-10T01:59:27.5393167Z Fork a new process to run a test 0 2025-10-10T01:59:27.5397409Z `transformers==4.56.2` installed, but `transformers<=4.48` is required to run this model. Reason: HF model is not compatible. 2025-10-10T01:59:27.8431020Z PASSED 2025-10-10T01:59:27.8561562Z models/test_initialization.py::test_can_initialize_large_subset[KimiVLForConditionalGeneration] Fork a new process to run a test 11305 2025-10-10T01:59:27.8573220Z Fork a new process to run a test 0 2025-10-10T01:59:27.8849536Z INFO 10-10 01:59:27 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='KimiVLForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'moonshotai/Kimi-VL-A3B-Instruct'} 2025-10-10T01:59:28.0080137Z 2025-10-10T01:59:28.0082571Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:59:28.0082885Z config.json: 2.00kB [00:00, 8.52MB/s] 2025-10-10T01:59:28.1186848Z 2025-10-10T01:59:28.1188585Z preprocessor_config.json: 0% 0.00/401 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:59:36.8810956Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] EngineCore failed to start. 2025-10-10T01:59:36.8811698Z 2025-10-10T01:59:36.8812187Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] Traceback (most recent call last): 2025-10-10T01:59:36.8812601Z 2025-10-10T01:59:36.8813756Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:36.8814418Z 2025-10-10T01:59:36.8814842Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:36.8815234Z 2025-10-10T01:59:36.8815587Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8815932Z 2025-10-10T01:59:36.8816556Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:36.8817110Z 2025-10-10T01:59:36.8817529Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:36.8817931Z 2025-10-10T01:59:36.8818752Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:36.8819416Z 2025-10-10T01:59:36.8819839Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:36.8820231Z 2025-10-10T01:59:36.8820579Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8820933Z 2025-10-10T01:59:36.8821599Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:36.8822182Z 2025-10-10T01:59:36.8822649Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] self._init_executor() 2025-10-10T01:59:36.8823152Z 2025-10-10T01:59:36.8824104Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:36.8824890Z 2025-10-10T01:59:36.8825546Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:36.8826136Z 2025-10-10T01:59:36.8827058Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:36.8827728Z 2025-10-10T01:59:36.8828115Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:36.8828471Z 2025-10-10T01:59:36.8828848Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8829212Z 2025-10-10T01:59:36.8829735Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:36.8830203Z 2025-10-10T01:59:36.8830483Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:59:36.8830759Z 2025-10-10T01:59:36.8831014Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8831292Z 2025-10-10T01:59:36.8831830Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:36.8832309Z 2025-10-10T01:59:36.8832611Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:36.8832906Z 2025-10-10T01:59:36.8833372Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8833666Z 2025-10-10T01:59:36.8834220Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:36.8834713Z 2025-10-10T01:59:36.8835023Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:59:36.8835336Z 2025-10-10T01:59:36.8835613Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8835893Z 2025-10-10T01:59:36.8836347Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:36.8836767Z 2025-10-10T01:59:36.8837123Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:36.8837458Z 2025-10-10T01:59:36.8837749Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8838037Z 2025-10-10T01:59:36.8838382Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:59:36.8838722Z 2025-10-10T01:59:36.8839199Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:59:36.8839552Z 2025-10-10T01:59:36.8839937Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:36.8840305Z 2025-10-10T01:59:36.8840654Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:59:36.8840990Z 2025-10-10T01:59:36.8841370Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:59:36.8841722Z 2025-10-10T01:59:36.8842102Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:36.8842520Z 2025-10-10T01:59:36.8843031Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:36.8843501Z 2025-10-10T01:59:36.8843905Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:36.8844284Z 2025-10-10T01:59:36.8844813Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:36.8845303Z 2025-10-10T01:59:36.8845678Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:36.8846060Z 2025-10-10T01:59:36.8846621Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:36.8847148Z 2025-10-10T01:59:36.8847449Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:59:36.8847750Z 2025-10-10T01:59:36.8848450Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:36.8849035Z 2025-10-10T01:59:36.8849416Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:36.8849772Z 2025-10-10T01:59:36.8850038Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8850304Z 2025-10-10T01:59:36.8850874Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:36.8851410Z 2025-10-10T01:59:36.8851713Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:59:36.8852018Z 2025-10-10T01:59:36.8852294Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8852573Z 2025-10-10T01:59:36.8853178Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:36.8853738Z 2025-10-10T01:59:36.8854021Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:59:36.8854318Z 2025-10-10T01:59:36.8854580Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8854849Z 2025-10-10T01:59:36.8855459Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:36.8856008Z 2025-10-10T01:59:36.8856331Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:36.8856647Z 2025-10-10T01:59:36.8856928Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8857202Z 2025-10-10T01:59:36.8857734Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:36.8858282Z 2025-10-10T01:59:36.8858575Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:59:36.8858867Z 2025-10-10T01:59:36.8859222Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8859539Z 2025-10-10T01:59:36.8860073Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:36.8860566Z 2025-10-10T01:59:36.8860878Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:59:36.8861192Z 2025-10-10T01:59:36.8861419Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:59:36.8861672Z 2025-10-10T01:59:36.8862155Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:36.8862607Z 2025-10-10T01:59:36.8862864Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] raise RuntimeError( 2025-10-10T01:59:36.8863138Z 2025-10-10T01:59:36.8863758Z (EngineCore_DP0 pid=11383) ERROR 10-10 01:59:36 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:36.8864467Z (EngineCore_DP0 pid=11383) Process EngineCore_DP0: 2025-10-10T01:59:36.8864871Z (EngineCore_DP0 pid=11383) Traceback (most recent call last): 2025-10-10T01:59:36.8865487Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:59:36.8866020Z (EngineCore_DP0 pid=11383) self.run() 2025-10-10T01:59:36.8866575Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:59:36.8867158Z (EngineCore_DP0 pid=11383) self._target(*self._args, **self._kwargs) 2025-10-10T01:59:36.8867834Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:59:36.8868388Z (EngineCore_DP0 pid=11383) raise e 2025-10-10T01:59:36.8868992Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:36.8869638Z (EngineCore_DP0 pid=11383) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:36.8870096Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8870733Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:36.8871472Z (EngineCore_DP0 pid=11383) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:36.8872266Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:36.8873033Z (EngineCore_DP0 pid=11383) self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:36.8873562Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8874216Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:36.8874810Z (EngineCore_DP0 pid=11383) self._init_executor() 2025-10-10T01:59:36.8875530Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:36.8876265Z (EngineCore_DP0 pid=11383) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:36.8877047Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:36.8877769Z (EngineCore_DP0 pid=11383) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:36.8878266Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8878909Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:36.8879565Z (EngineCore_DP0 pid=11383) return func(*args, **kwargs) 2025-10-10T01:59:36.8879965Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8880629Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:36.8881258Z (EngineCore_DP0 pid=11383) worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:36.8881743Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8882419Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:36.8883084Z (EngineCore_DP0 pid=11383) module = importlib.import_module(module_name) 2025-10-10T01:59:36.8883533Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8884118Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:36.8884743Z (EngineCore_DP0 pid=11383) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:36.8885252Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8885772Z (EngineCore_DP0 pid=11383) File "", line 1387, in _gcd_import 2025-10-10T01:59:36.8886339Z (EngineCore_DP0 pid=11383) File "", line 1360, in _find_and_load 2025-10-10T01:59:36.8886939Z (EngineCore_DP0 pid=11383) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:36.8887527Z (EngineCore_DP0 pid=11383) File "", line 935, in _load_unlocked 2025-10-10T01:59:36.8888112Z (EngineCore_DP0 pid=11383) File "", line 999, in exec_module 2025-10-10T01:59:36.8888724Z (EngineCore_DP0 pid=11383) File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:36.8889469Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:36.8890160Z (EngineCore_DP0 pid=11383) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:36.8890893Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:36.8891758Z (EngineCore_DP0 pid=11383) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:36.8892558Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:36.8893309Z (EngineCore_DP0 pid=11383) class FlashAttentionMetadataBuilder( 2025-10-10T01:59:36.8894136Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:36.8894997Z (EngineCore_DP0 pid=11383) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:36.8895479Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8896366Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:36.8897063Z (EngineCore_DP0 pid=11383) if not is_fa_version_supported(fa_version): 2025-10-10T01:59:36.8897507Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8898260Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:36.8898978Z (EngineCore_DP0 pid=11383) return _is_fa2_supported(device)[0] 2025-10-10T01:59:36.8899396Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8900208Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:36.8900970Z (EngineCore_DP0 pid=11383) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:36.8901527Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8902328Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:36.8903002Z (EngineCore_DP0 pid=11383) prop = get_device_properties(device) 2025-10-10T01:59:36.8903436Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:36.8904104Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:36.8904776Z (EngineCore_DP0 pid=11383) _lazy_init() # will define _get_device_properties 2025-10-10T01:59:36.8905176Z (EngineCore_DP0 pid=11383) ^^^^^^^^^^^^ 2025-10-10T01:59:36.8905754Z (EngineCore_DP0 pid=11383) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:36.8906325Z (EngineCore_DP0 pid=11383) raise RuntimeError( 2025-10-10T01:59:36.8907018Z (EngineCore_DP0 pid=11383) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:37.3052132Z FAILED 2025-10-10T01:59:37.3181041Z models/test_initialization.py::test_can_initialize_large_subset[Ovis2_5] Fork a new process to run a test 11387 2025-10-10T01:59:37.3192025Z Fork a new process to run a test 0 2025-10-10T01:59:37.3471811Z INFO 10-10 01:59:37 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Ovis2_5', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'AIDC-AI/Ovis2.5-2B'} 2025-10-10T01:59:37.5021168Z 2025-10-10T01:59:37.5023438Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:59:37.5023770Z config.json: 2.00kB [00:00, 10.4MB/s] 2025-10-10T01:59:37.5876353Z 2025-10-10T01:59:37.5877503Z configuration_ovis2_5.py: 0.00B [00:00, ?B/s] 2025-10-10T01:59:37.5877871Z configuration_ovis2_5.py: 3.94kB [00:00, 38.2MB/s] 2025-10-10T01:59:37.5971033Z A new version of the following files was downloaded from https://huggingface.co/AIDC-AI/Ovis2.5-2B: 2025-10-10T01:59:37.5971551Z - configuration_ovis2_5.py 2025-10-10T01:59:37.5972339Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:59:37.8818772Z 2025-10-10T01:59:37.8819234Z preprocessor_config.json: 0% 0.00/394 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:59:46.5724473Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] EngineCore failed to start. 2025-10-10T01:59:46.5724907Z 2025-10-10T01:59:46.5725327Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] Traceback (most recent call last): 2025-10-10T01:59:46.5725714Z 2025-10-10T01:59:46.5726404Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:46.5727019Z 2025-10-10T01:59:46.5727671Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:46.5728076Z 2025-10-10T01:59:46.5728437Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5728787Z 2025-10-10T01:59:46.5729413Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:46.5729968Z 2025-10-10T01:59:46.5730397Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:46.5730785Z 2025-10-10T01:59:46.5731284Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:46.5731743Z 2025-10-10T01:59:46.5732063Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:46.5732375Z 2025-10-10T01:59:46.5732658Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5732933Z 2025-10-10T01:59:46.5733441Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:46.5733910Z 2025-10-10T01:59:46.5734166Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] self._init_executor() 2025-10-10T01:59:46.5734434Z 2025-10-10T01:59:46.5734975Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:46.5735522Z 2025-10-10T01:59:46.5736026Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:46.5736646Z 2025-10-10T01:59:46.5737506Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:46.5738355Z 2025-10-10T01:59:46.5738754Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:46.5739252Z 2025-10-10T01:59:46.5739566Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5739870Z 2025-10-10T01:59:46.5740485Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:46.5741060Z 2025-10-10T01:59:46.5741365Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:59:46.5741648Z 2025-10-10T01:59:46.5741903Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5742172Z 2025-10-10T01:59:46.5742704Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:46.5743202Z 2025-10-10T01:59:46.5743497Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:46.5743792Z 2025-10-10T01:59:46.5744063Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5744335Z 2025-10-10T01:59:46.5744905Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:46.5745402Z 2025-10-10T01:59:46.5745710Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:59:46.5746011Z 2025-10-10T01:59:46.5746289Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5746561Z 2025-10-10T01:59:46.5747003Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:46.5747414Z 2025-10-10T01:59:46.5747755Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:46.5748094Z 2025-10-10T01:59:46.5748380Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5748666Z 2025-10-10T01:59:46.5749008Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:59:46.5749344Z 2025-10-10T01:59:46.5749698Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:59:46.5750040Z 2025-10-10T01:59:46.5750429Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:46.5750789Z 2025-10-10T01:59:46.5751161Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:59:46.5751514Z 2025-10-10T01:59:46.5751896Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:59:46.5752249Z 2025-10-10T01:59:46.5752629Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:46.5752992Z 2025-10-10T01:59:46.5753498Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:46.5754018Z 2025-10-10T01:59:46.5754374Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:46.5754716Z 2025-10-10T01:59:46.5755275Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:46.5755801Z 2025-10-10T01:59:46.5756178Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:46.5756544Z 2025-10-10T01:59:46.5757094Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:46.5757601Z 2025-10-10T01:59:46.5757902Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:59:46.5758198Z 2025-10-10T01:59:46.5758852Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:46.5759548Z 2025-10-10T01:59:46.5759999Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:46.5760374Z 2025-10-10T01:59:46.5760645Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5760907Z 2025-10-10T01:59:46.5761472Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:46.5761999Z 2025-10-10T01:59:46.5762296Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:59:46.5762608Z 2025-10-10T01:59:46.5762885Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5763166Z 2025-10-10T01:59:46.5763764Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:46.5764361Z 2025-10-10T01:59:46.5764649Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:59:46.5764941Z 2025-10-10T01:59:46.5765209Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5765480Z 2025-10-10T01:59:46.5766068Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:46.5766601Z 2025-10-10T01:59:46.5766928Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:46.5767247Z 2025-10-10T01:59:46.5767525Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5767805Z 2025-10-10T01:59:46.5768329Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:46.5768825Z 2025-10-10T01:59:46.5769105Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:59:46.5769458Z 2025-10-10T01:59:46.5769729Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5770003Z 2025-10-10T01:59:46.5770571Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:46.5771095Z 2025-10-10T01:59:46.5771419Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:59:46.5771724Z 2025-10-10T01:59:46.5771957Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:59:46.5772202Z 2025-10-10T01:59:46.5772690Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:46.5773142Z 2025-10-10T01:59:46.5773387Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] raise RuntimeError( 2025-10-10T01:59:46.5773652Z 2025-10-10T01:59:46.5774227Z (EngineCore_DP0 pid=11465) ERROR 10-10 01:59:46 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:46.5774967Z (EngineCore_DP0 pid=11465) Process EngineCore_DP0: 2025-10-10T01:59:46.5775385Z (EngineCore_DP0 pid=11465) Traceback (most recent call last): 2025-10-10T01:59:46.5775996Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:59:46.5776515Z (EngineCore_DP0 pid=11465) self.run() 2025-10-10T01:59:46.5777043Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:59:46.5777610Z (EngineCore_DP0 pid=11465) self._target(*self._args, **self._kwargs) 2025-10-10T01:59:46.5778283Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:59:46.5778835Z (EngineCore_DP0 pid=11465) raise e 2025-10-10T01:59:46.5779425Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:46.5780060Z (EngineCore_DP0 pid=11465) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:46.5780514Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5781143Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:46.5781790Z (EngineCore_DP0 pid=11465) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:46.5782455Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:46.5783079Z (EngineCore_DP0 pid=11465) self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:46.5783537Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5784198Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:46.5784792Z (EngineCore_DP0 pid=11465) self._init_executor() 2025-10-10T01:59:46.5785465Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:46.5786209Z (EngineCore_DP0 pid=11465) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:46.5786939Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:46.5787686Z (EngineCore_DP0 pid=11465) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:46.5788227Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5788873Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:46.5789463Z (EngineCore_DP0 pid=11465) return func(*args, **kwargs) 2025-10-10T01:59:46.5789853Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5790503Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:46.5791145Z (EngineCore_DP0 pid=11465) worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:46.5791582Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5792301Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:46.5792965Z (EngineCore_DP0 pid=11465) module = importlib.import_module(module_name) 2025-10-10T01:59:46.5793406Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5794011Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:46.5794642Z (EngineCore_DP0 pid=11465) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:46.5795131Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5795633Z (EngineCore_DP0 pid=11465) File "", line 1387, in _gcd_import 2025-10-10T01:59:46.5796407Z (EngineCore_DP0 pid=11465) File "", line 1360, in _find_and_load 2025-10-10T01:59:46.5797014Z (EngineCore_DP0 pid=11465) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:46.5797601Z (EngineCore_DP0 pid=11465) File "", line 935, in _load_unlocked 2025-10-10T01:59:46.5798172Z (EngineCore_DP0 pid=11465) File "", line 999, in exec_module 2025-10-10T01:59:46.5798779Z (EngineCore_DP0 pid=11465) File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:46.5799566Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:46.5800252Z (EngineCore_DP0 pid=11465) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:46.5800990Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:46.5801716Z (EngineCore_DP0 pid=11465) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:46.5802505Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:46.5803192Z (EngineCore_DP0 pid=11465) class FlashAttentionMetadataBuilder( 2025-10-10T01:59:46.5803988Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:46.5804907Z (EngineCore_DP0 pid=11465) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:46.5805452Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5806199Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:46.5806888Z (EngineCore_DP0 pid=11465) if not is_fa_version_supported(fa_version): 2025-10-10T01:59:46.5807336Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5808091Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:46.5808797Z (EngineCore_DP0 pid=11465) return _is_fa2_supported(device)[0] 2025-10-10T01:59:46.5809217Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5809939Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:46.5810725Z (EngineCore_DP0 pid=11465) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:46.5811193Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5811867Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:46.5812505Z (EngineCore_DP0 pid=11465) prop = get_device_properties(device) 2025-10-10T01:59:46.5812920Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:46.5813581Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:46.5814248Z (EngineCore_DP0 pid=11465) _lazy_init() # will define _get_device_properties 2025-10-10T01:59:46.5814649Z (EngineCore_DP0 pid=11465) ^^^^^^^^^^^^ 2025-10-10T01:59:46.5815234Z (EngineCore_DP0 pid=11465) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:46.5815800Z (EngineCore_DP0 pid=11465) raise RuntimeError( 2025-10-10T01:59:46.5816487Z (EngineCore_DP0 pid=11465) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:47.0004897Z FAILED 2025-10-10T01:59:47.0133967Z models/test_initialization.py::test_can_initialize_large_subset[Phi4MMForCausalLM] Fork a new process to run a test 11469 2025-10-10T01:59:47.0144911Z Fork a new process to run a test 0 2025-10-10T01:59:47.0417230Z INFO 10-10 01:59:47 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Phi4MMForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'microsoft/Phi-4-multimodal-instruct'} 2025-10-10T01:59:47.1716609Z 2025-10-10T01:59:47.1719366Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T01:59:47.1719667Z config.json: 4.63kB [00:00, 19.1MB/s] 2025-10-10T01:59:47.2722660Z 2025-10-10T01:59:47.2725330Z configuration_phi4mm.py: 0.00B [00:00, ?B/s] 2025-10-10T01:59:47.2725718Z configuration_phi4mm.py: 11.0kB [00:00, 29.6MB/s] 2025-10-10T01:59:47.2818355Z A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-4-multimodal-instruct: 2025-10-10T01:59:47.2819138Z - configuration_phi4mm.py 2025-10-10T01:59:47.2819759Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T01:59:47.3162264Z INFO 10-10 01:59:47 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T01:59:47.5127735Z 2025-10-10T01:59:47.5128343Z preprocessor_config.json: 0% 0.00/482 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T01:59:56.2513073Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] EngineCore failed to start. 2025-10-10T01:59:56.2513470Z 2025-10-10T01:59:56.2514129Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] Traceback (most recent call last): 2025-10-10T01:59:56.2514508Z 2025-10-10T01:59:56.2515354Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:56.2515967Z 2025-10-10T01:59:56.2516620Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:56.2517029Z 2025-10-10T01:59:56.2517396Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2517755Z 2025-10-10T01:59:56.2518381Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:56.2518971Z 2025-10-10T01:59:56.2519521Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:56.2519935Z 2025-10-10T01:59:56.2520510Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:56.2521097Z 2025-10-10T01:59:56.2521441Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:56.2521758Z 2025-10-10T01:59:56.2522045Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2522323Z 2025-10-10T01:59:56.2522846Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:56.2523324Z 2025-10-10T01:59:56.2523584Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] self._init_executor() 2025-10-10T01:59:56.2523957Z 2025-10-10T01:59:56.2524632Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:56.2525252Z 2025-10-10T01:59:56.2525780Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:56.2526114Z 2025-10-10T01:59:56.2526877Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:56.2527494Z 2025-10-10T01:59:56.2527943Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:56.2528520Z 2025-10-10T01:59:56.2528840Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2529254Z 2025-10-10T01:59:56.2529865Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:56.2530407Z 2025-10-10T01:59:56.2530684Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] return func(*args, **kwargs) 2025-10-10T01:59:56.2530959Z 2025-10-10T01:59:56.2531221Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2531484Z 2025-10-10T01:59:56.2532014Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:56.2532494Z 2025-10-10T01:59:56.2532790Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:56.2533078Z 2025-10-10T01:59:56.2533348Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2533627Z 2025-10-10T01:59:56.2534199Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:56.2534698Z 2025-10-10T01:59:56.2535015Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T01:59:56.2535321Z 2025-10-10T01:59:56.2535593Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2535880Z 2025-10-10T01:59:56.2536324Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:56.2536736Z 2025-10-10T01:59:56.2537084Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:56.2537419Z 2025-10-10T01:59:56.2537711Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2537999Z 2025-10-10T01:59:56.2538353Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T01:59:56.2538685Z 2025-10-10T01:59:56.2539041Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T01:59:56.2539386Z 2025-10-10T01:59:56.2539760Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:56.2540124Z 2025-10-10T01:59:56.2540469Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T01:59:56.2540808Z 2025-10-10T01:59:56.2541168Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "", line 999, in exec_module 2025-10-10T01:59:56.2541524Z 2025-10-10T01:59:56.2541896Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:56.2542256Z 2025-10-10T01:59:56.2542760Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:56.2543328Z 2025-10-10T01:59:56.2543690Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:56.2544026Z 2025-10-10T01:59:56.2544600Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:56.2545117Z 2025-10-10T01:59:56.2545507Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:56.2545869Z 2025-10-10T01:59:56.2546418Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:56.2546943Z 2025-10-10T01:59:56.2547242Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T01:59:56.2547543Z 2025-10-10T01:59:56.2548169Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:56.2548793Z 2025-10-10T01:59:56.2549276Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:56.2549699Z 2025-10-10T01:59:56.2550010Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2550320Z 2025-10-10T01:59:56.2550996Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:56.2551519Z 2025-10-10T01:59:56.2551818Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T01:59:56.2552121Z 2025-10-10T01:59:56.2552403Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2552678Z 2025-10-10T01:59:56.2553277Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:56.2553836Z 2025-10-10T01:59:56.2554116Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T01:59:56.2554403Z 2025-10-10T01:59:56.2554662Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2554937Z 2025-10-10T01:59:56.2555513Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:56.2556059Z 2025-10-10T01:59:56.2556385Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:56.2556700Z 2025-10-10T01:59:56.2556980Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2557252Z 2025-10-10T01:59:56.2557781Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:56.2558268Z 2025-10-10T01:59:56.2558553Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] prop = get_device_properties(device) 2025-10-10T01:59:56.2558888Z 2025-10-10T01:59:56.2559274Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2559556Z 2025-10-10T01:59:56.2560128Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:56.2560656Z 2025-10-10T01:59:56.2560974Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T01:59:56.2561285Z 2025-10-10T01:59:56.2561510Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T01:59:56.2561760Z 2025-10-10T01:59:56.2562242Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:56.2562693Z 2025-10-10T01:59:56.2562943Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] raise RuntimeError( 2025-10-10T01:59:56.2563201Z 2025-10-10T01:59:56.2563777Z (EngineCore_DP0 pid=11548) ERROR 10-10 01:59:56 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:56.2564548Z (EngineCore_DP0 pid=11548) Process EngineCore_DP0: 2025-10-10T01:59:56.2564948Z (EngineCore_DP0 pid=11548) Traceback (most recent call last): 2025-10-10T01:59:56.2565585Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T01:59:56.2566105Z (EngineCore_DP0 pid=11548) self.run() 2025-10-10T01:59:56.2566637Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T01:59:56.2567208Z (EngineCore_DP0 pid=11548) self._target(*self._args, **self._kwargs) 2025-10-10T01:59:56.2567890Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T01:59:56.2568442Z (EngineCore_DP0 pid=11548) raise e 2025-10-10T01:59:56.2569138Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T01:59:56.2569921Z (EngineCore_DP0 pid=11548) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T01:59:56.2570447Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2571092Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T01:59:56.2571736Z (EngineCore_DP0 pid=11548) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T01:59:56.2572420Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T01:59:56.2573046Z (EngineCore_DP0 pid=11548) self.model_executor = executor_class(vllm_config) 2025-10-10T01:59:56.2573508Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2574163Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T01:59:56.2574755Z (EngineCore_DP0 pid=11548) self._init_executor() 2025-10-10T01:59:56.2575418Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T01:59:56.2576163Z (EngineCore_DP0 pid=11548) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T01:59:56.2576892Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T01:59:56.2577660Z (EngineCore_DP0 pid=11548) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T01:59:56.2578204Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2578840Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T01:59:56.2579429Z (EngineCore_DP0 pid=11548) return func(*args, **kwargs) 2025-10-10T01:59:56.2579813Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2580453Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T01:59:56.2581087Z (EngineCore_DP0 pid=11548) worker_class = resolve_obj_by_qualname( 2025-10-10T01:59:56.2581512Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2589085Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T01:59:56.2589915Z (EngineCore_DP0 pid=11548) module = importlib.import_module(module_name) 2025-10-10T01:59:56.2590399Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2591019Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T01:59:56.2591695Z (EngineCore_DP0 pid=11548) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T01:59:56.2592208Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2592724Z (EngineCore_DP0 pid=11548) File "", line 1387, in _gcd_import 2025-10-10T01:59:56.2593293Z (EngineCore_DP0 pid=11548) File "", line 1360, in _find_and_load 2025-10-10T01:59:56.2593897Z (EngineCore_DP0 pid=11548) File "", line 1331, in _find_and_load_unlocked 2025-10-10T01:59:56.2594484Z (EngineCore_DP0 pid=11548) File "", line 935, in _load_unlocked 2025-10-10T01:59:56.2595062Z (EngineCore_DP0 pid=11548) File "", line 999, in exec_module 2025-10-10T01:59:56.2595679Z (EngineCore_DP0 pid=11548) File "", line 488, in _call_with_frames_removed 2025-10-10T01:59:56.2596723Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T01:59:56.2597430Z (EngineCore_DP0 pid=11548) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T01:59:56.2598171Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T01:59:56.2598904Z (EngineCore_DP0 pid=11548) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T01:59:56.2599785Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T01:59:56.2600491Z (EngineCore_DP0 pid=11548) class FlashAttentionMetadataBuilder( 2025-10-10T01:59:56.2601294Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T01:59:56.2602264Z (EngineCore_DP0 pid=11548) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T01:59:56.2602816Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2603581Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T01:59:56.2604272Z (EngineCore_DP0 pid=11548) if not is_fa_version_supported(fa_version): 2025-10-10T01:59:56.2604728Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2605487Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T01:59:56.2606194Z (EngineCore_DP0 pid=11548) return _is_fa2_supported(device)[0] 2025-10-10T01:59:56.2606609Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2607329Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T01:59:56.2608120Z (EngineCore_DP0 pid=11548) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T01:59:56.2608587Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2609385Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T01:59:56.2610146Z (EngineCore_DP0 pid=11548) prop = get_device_properties(device) 2025-10-10T01:59:56.2610648Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T01:59:56.2611390Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T01:59:56.2612081Z (EngineCore_DP0 pid=11548) _lazy_init() # will define _get_device_properties 2025-10-10T01:59:56.2612486Z (EngineCore_DP0 pid=11548) ^^^^^^^^^^^^ 2025-10-10T01:59:56.2613071Z (EngineCore_DP0 pid=11548) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T01:59:56.2613640Z (EngineCore_DP0 pid=11548) raise RuntimeError( 2025-10-10T01:59:56.2614334Z (EngineCore_DP0 pid=11548) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T01:59:56.6809913Z FAILED 2025-10-10T01:59:56.6938545Z models/test_initialization.py::test_can_initialize_large_subset[H2OVLChatModel] Fork a new process to run a test 11552 2025-10-10T01:59:56.6950199Z Fork a new process to run a test 0 2025-10-10T01:59:56.6954085Z `transformers==4.56.2` installed, but `transformers<=4.48` is required to run this model. Reason: HF model is not compatible. 2025-10-10T01:59:57.0022597Z PASSED 2025-10-10T01:59:57.0150692Z models/test_initialization.py::test_can_initialize_large_subset[Qwen3MoeForCausalLM] Fork a new process to run a test 11553 2025-10-10T01:59:57.0162026Z Fork a new process to run a test 0 2025-10-10T01:59:57.0439324Z INFO 10-10 01:59:57 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen3MoeForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Qwen/Qwen3-30B-A3B'} 2025-10-10T01:59:57.1683853Z 2025-10-10T01:59:57.1684937Z config.json: 0% 0.00/963 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:06.1143792Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:06.1145341Z 2025-10-10T02:00:06.1145772Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:06.1146145Z 2025-10-10T02:00:06.1146956Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:06.1147691Z 2025-10-10T02:00:06.1148095Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:06.1148437Z 2025-10-10T02:00:06.1148721Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1149029Z 2025-10-10T02:00:06.1149525Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:06.1150140Z 2025-10-10T02:00:06.1150495Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:06.1150816Z 2025-10-10T02:00:06.1151308Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:06.1151749Z 2025-10-10T02:00:06.1152181Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:06.1152504Z 2025-10-10T02:00:06.1152796Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1153173Z 2025-10-10T02:00:06.1154086Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:06.1154866Z 2025-10-10T02:00:06.1155193Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] self._init_executor() 2025-10-10T02:00:06.1155628Z 2025-10-10T02:00:06.1156563Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:06.1157301Z 2025-10-10T02:00:06.1157685Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:06.1158064Z 2025-10-10T02:00:06.1158609Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:06.1159245Z 2025-10-10T02:00:06.1159598Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:06.1159936Z 2025-10-10T02:00:06.1160226Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1160513Z 2025-10-10T02:00:06.1161002Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:06.1161467Z 2025-10-10T02:00:06.1161736Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:06.1162004Z 2025-10-10T02:00:06.1162260Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1162517Z 2025-10-10T02:00:06.1163054Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:06.1163629Z 2025-10-10T02:00:06.1163940Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:06.1164240Z 2025-10-10T02:00:06.1164565Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1164889Z 2025-10-10T02:00:06.1165441Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:06.1165951Z 2025-10-10T02:00:06.1166258Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:06.1166565Z 2025-10-10T02:00:06.1166839Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1167121Z 2025-10-10T02:00:06.1167559Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:06.1167979Z 2025-10-10T02:00:06.1168330Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:06.1168666Z 2025-10-10T02:00:06.1169041Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1169329Z 2025-10-10T02:00:06.1169676Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:06.1170005Z 2025-10-10T02:00:06.1170363Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:06.1170709Z 2025-10-10T02:00:06.1171083Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:06.1171445Z 2025-10-10T02:00:06.1171789Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:06.1172128Z 2025-10-10T02:00:06.1172492Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:06.1172846Z 2025-10-10T02:00:06.1173214Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:06.1173582Z 2025-10-10T02:00:06.1174114Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:06.1174586Z 2025-10-10T02:00:06.1174944Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:06.1175278Z 2025-10-10T02:00:06.1175809Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:06.1176287Z 2025-10-10T02:00:06.1176667Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:06.1177026Z 2025-10-10T02:00:06.1177577Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:06.1178159Z 2025-10-10T02:00:06.1178461Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:06.1178764Z 2025-10-10T02:00:06.1179428Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:06.1180043Z 2025-10-10T02:00:06.1180413Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:06.1180769Z 2025-10-10T02:00:06.1181022Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1181284Z 2025-10-10T02:00:06.1181852Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:06.1182372Z 2025-10-10T02:00:06.1182678Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:06.1182970Z 2025-10-10T02:00:06.1183250Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1183526Z 2025-10-10T02:00:06.1184168Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:06.1184724Z 2025-10-10T02:00:06.1185009Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:06.1185302Z 2025-10-10T02:00:06.1185559Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1185838Z 2025-10-10T02:00:06.1186411Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:06.1186953Z 2025-10-10T02:00:06.1187278Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:06.1187597Z 2025-10-10T02:00:06.1187879Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1188157Z 2025-10-10T02:00:06.1188688Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:06.1189171Z 2025-10-10T02:00:06.1189457Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:06.1189747Z 2025-10-10T02:00:06.1190014Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1190299Z 2025-10-10T02:00:06.1190823Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:06.1191313Z 2025-10-10T02:00:06.1191624Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:06.1191934Z 2025-10-10T02:00:06.1192161Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:06.1192410Z 2025-10-10T02:00:06.1192890Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:06.1193396Z 2025-10-10T02:00:06.1193650Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] raise RuntimeError( 2025-10-10T02:00:06.1193907Z 2025-10-10T02:00:06.1194520Z (EngineCore_DP0 pid=11632) ERROR 10-10 02:00:06 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:06.1195262Z (EngineCore_DP0 pid=11632) Process EngineCore_DP0: 2025-10-10T02:00:06.1195654Z (EngineCore_DP0 pid=11632) Traceback (most recent call last): 2025-10-10T02:00:06.1196555Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:06.1197092Z (EngineCore_DP0 pid=11632) self.run() 2025-10-10T02:00:06.1197629Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:06.1198203Z (EngineCore_DP0 pid=11632) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:06.1198917Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:06.1199550Z (EngineCore_DP0 pid=11632) raise e 2025-10-10T02:00:06.1200700Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:06.1201398Z (EngineCore_DP0 pid=11632) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:06.1201854Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1202481Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:06.1203134Z (EngineCore_DP0 pid=11632) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:06.1203802Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:06.1204430Z (EngineCore_DP0 pid=11632) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:06.1204891Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1205541Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:06.1206130Z (EngineCore_DP0 pid=11632) self._init_executor() 2025-10-10T02:00:06.1206811Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:06.1207509Z (EngineCore_DP0 pid=11632) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:06.1208242Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:06.1208962Z (EngineCore_DP0 pid=11632) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:06.1209467Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1210108Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:06.1210698Z (EngineCore_DP0 pid=11632) return func(*args, **kwargs) 2025-10-10T02:00:06.1211081Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1211837Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:06.1212473Z (EngineCore_DP0 pid=11632) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:06.1212965Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1213700Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:06.1214359Z (EngineCore_DP0 pid=11632) module = importlib.import_module(module_name) 2025-10-10T02:00:06.1214807Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1215387Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:06.1216013Z (EngineCore_DP0 pid=11632) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:06.1216519Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1217022Z (EngineCore_DP0 pid=11632) File "", line 1387, in _gcd_import 2025-10-10T02:00:06.1217581Z (EngineCore_DP0 pid=11632) File "", line 1360, in _find_and_load 2025-10-10T02:00:06.1218227Z (EngineCore_DP0 pid=11632) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:06.1218816Z (EngineCore_DP0 pid=11632) File "", line 935, in _load_unlocked 2025-10-10T02:00:06.1219389Z (EngineCore_DP0 pid=11632) File "", line 999, in exec_module 2025-10-10T02:00:06.1219995Z (EngineCore_DP0 pid=11632) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:06.1220736Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:06.1221416Z (EngineCore_DP0 pid=11632) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:06.1222147Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:06.1222876Z (EngineCore_DP0 pid=11632) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:06.1223661Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:06.1224338Z (EngineCore_DP0 pid=11632) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:06.1225130Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:06.1225946Z (EngineCore_DP0 pid=11632) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:06.1226448Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1227150Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:06.1227840Z (EngineCore_DP0 pid=11632) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:06.1228293Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1229038Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:06.1229813Z (EngineCore_DP0 pid=11632) return _is_fa2_supported(device)[0] 2025-10-10T02:00:06.1230233Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1231017Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:06.1231829Z (EngineCore_DP0 pid=11632) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:06.1232284Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1232959Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:06.1233602Z (EngineCore_DP0 pid=11632) prop = get_device_properties(device) 2025-10-10T02:00:06.1234026Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:06.1234689Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:06.1235351Z (EngineCore_DP0 pid=11632) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:06.1235755Z (EngineCore_DP0 pid=11632) ^^^^^^^^^^^^ 2025-10-10T02:00:06.1236381Z (EngineCore_DP0 pid=11632) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:06.1236954Z (EngineCore_DP0 pid=11632) raise RuntimeError( 2025-10-10T02:00:06.1237641Z (EngineCore_DP0 pid=11632) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:06.5307221Z FAILED 2025-10-10T02:00:06.5436186Z models/test_initialization.py::test_can_initialize_large_subset[RobertaForMaskedLM] Fork a new process to run a test 11636 2025-10-10T02:00:06.5446965Z Fork a new process to run a test 0 2025-10-10T02:00:06.5723006Z INFO 10-10 02:00:06 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='RobertaForMaskedLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'sentence-transformers/all-roberta-large-v1'} 2025-10-10T02:00:07.6593534Z 2025-10-10T02:00:07.6593847Z config.json: 0% 0.00/650 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 256, 'local_cache_dir': None} 2025-10-10T02:00:10.1046743Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:10.1047314Z 2025-10-10T02:00:10.1048214Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:10.1048607Z 2025-10-10T02:00:10.1049507Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:10.1050232Z 2025-10-10T02:00:10.1050653Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:10.1051041Z 2025-10-10T02:00:10.1051395Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1051751Z 2025-10-10T02:00:10.1052304Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:10.1052865Z 2025-10-10T02:00:10.1053258Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:10.1053656Z 2025-10-10T02:00:10.1054266Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:10.1054735Z 2025-10-10T02:00:10.1055170Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:10.1055499Z 2025-10-10T02:00:10.1055807Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1056087Z 2025-10-10T02:00:10.1056745Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:10.1057264Z 2025-10-10T02:00:10.1057744Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] self._init_executor() 2025-10-10T02:00:10.1058024Z 2025-10-10T02:00:10.1058718Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:10.1059233Z 2025-10-10T02:00:10.1059716Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:10.1060056Z 2025-10-10T02:00:10.1060748Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:10.1061379Z 2025-10-10T02:00:10.1061748Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:10.1062217Z 2025-10-10T02:00:10.1062532Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1062816Z 2025-10-10T02:00:10.1063312Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:10.1063772Z 2025-10-10T02:00:10.1064060Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:10.1064334Z 2025-10-10T02:00:10.1064588Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1064851Z 2025-10-10T02:00:10.1065366Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:10.1065920Z 2025-10-10T02:00:10.1066212Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:10.1066514Z 2025-10-10T02:00:10.1066826Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1067150Z 2025-10-10T02:00:10.1067688Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:10.1068183Z 2025-10-10T02:00:10.1068485Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:10.1068792Z 2025-10-10T02:00:10.1069063Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1069341Z 2025-10-10T02:00:10.1069791Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:10.1070202Z 2025-10-10T02:00:10.1070583Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:10.1070922Z 2025-10-10T02:00:10.1071269Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1071561Z 2025-10-10T02:00:10.1071913Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:10.1072245Z 2025-10-10T02:00:10.1072600Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:10.1072940Z 2025-10-10T02:00:10.1073321Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:10.1073679Z 2025-10-10T02:00:10.1074030Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:10.1074370Z 2025-10-10T02:00:10.1074733Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:10.1075092Z 2025-10-10T02:00:10.1075475Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:10.1075845Z 2025-10-10T02:00:10.1076352Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:10.1076829Z 2025-10-10T02:00:10.1077182Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:10.1077522Z 2025-10-10T02:00:10.1078044Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:10.1078543Z 2025-10-10T02:00:10.1078919Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:10.1079425Z 2025-10-10T02:00:10.1079989Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:10.1080514Z 2025-10-10T02:00:10.1080881Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:10.1081188Z 2025-10-10T02:00:10.1081906Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:10.1082544Z 2025-10-10T02:00:10.1082920Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:10.1083283Z 2025-10-10T02:00:10.1083542Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1083811Z 2025-10-10T02:00:10.1084378Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:10.1084911Z 2025-10-10T02:00:10.1085209Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:10.1085513Z 2025-10-10T02:00:10.1085788Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1086062Z 2025-10-10T02:00:10.1086707Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:10.1087263Z 2025-10-10T02:00:10.1087554Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:10.1087843Z 2025-10-10T02:00:10.1088188Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1088607Z 2025-10-10T02:00:10.1089232Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:10.1089788Z 2025-10-10T02:00:10.1090120Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:10.1090446Z 2025-10-10T02:00:10.1090723Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1091005Z 2025-10-10T02:00:10.1091539Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:10.1092039Z 2025-10-10T02:00:10.1092324Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:10.1092616Z 2025-10-10T02:00:10.1092892Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1093167Z 2025-10-10T02:00:10.1093704Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:10.1094198Z 2025-10-10T02:00:10.1094518Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:10.1094823Z 2025-10-10T02:00:10.1095051Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:10.1095300Z 2025-10-10T02:00:10.1095785Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:10.1096540Z 2025-10-10T02:00:10.1096813Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] raise RuntimeError( 2025-10-10T02:00:10.1097084Z 2025-10-10T02:00:10.1097752Z (EngineCore_DP0 pid=11644) ERROR 10-10 02:00:10 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:10.1098555Z (EngineCore_DP0 pid=11644) Process EngineCore_DP0: 2025-10-10T02:00:10.1098965Z (EngineCore_DP0 pid=11644) Traceback (most recent call last): 2025-10-10T02:00:10.1099588Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:10.1100098Z (EngineCore_DP0 pid=11644) self.run() 2025-10-10T02:00:10.1100631Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:10.1101199Z (EngineCore_DP0 pid=11644) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:10.1101884Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:10.1102436Z (EngineCore_DP0 pid=11644) raise e 2025-10-10T02:00:10.1103106Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:10.1103756Z (EngineCore_DP0 pid=11644) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:10.1104213Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1104843Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:10.1105492Z (EngineCore_DP0 pid=11644) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:10.1106166Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:10.1106800Z (EngineCore_DP0 pid=11644) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:10.1107263Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1107921Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:10.1108518Z (EngineCore_DP0 pid=11644) self._init_executor() 2025-10-10T02:00:10.1109195Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:10.1109898Z (EngineCore_DP0 pid=11644) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:10.1110644Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:10.1111385Z (EngineCore_DP0 pid=11644) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:10.1111893Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1112531Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:10.1113124Z (EngineCore_DP0 pid=11644) return func(*args, **kwargs) 2025-10-10T02:00:10.1113515Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1114159Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:10.1114892Z (EngineCore_DP0 pid=11644) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:10.1115364Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1116081Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:10.1116749Z (EngineCore_DP0 pid=11644) module = importlib.import_module(module_name) 2025-10-10T02:00:10.1117198Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1117809Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:10.1118442Z (EngineCore_DP0 pid=11644) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:10.1118943Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1119535Z (EngineCore_DP0 pid=11644) File "", line 1387, in _gcd_import 2025-10-10T02:00:10.1120104Z (EngineCore_DP0 pid=11644) File "", line 1360, in _find_and_load 2025-10-10T02:00:10.1120745Z (EngineCore_DP0 pid=11644) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:10.1121338Z (EngineCore_DP0 pid=11644) File "", line 935, in _load_unlocked 2025-10-10T02:00:10.1121937Z (EngineCore_DP0 pid=11644) File "", line 999, in exec_module 2025-10-10T02:00:10.1122545Z (EngineCore_DP0 pid=11644) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:10.1123287Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:10.1123974Z (EngineCore_DP0 pid=11644) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:10.1124702Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:10.1125425Z (EngineCore_DP0 pid=11644) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:10.1126212Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:10.1126878Z (EngineCore_DP0 pid=11644) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:10.1127670Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:10.1128492Z (EngineCore_DP0 pid=11644) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:10.1128972Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1129669Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:10.1130349Z (EngineCore_DP0 pid=11644) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:10.1130794Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1131544Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:10.1132306Z (EngineCore_DP0 pid=11644) return _is_fa2_supported(device)[0] 2025-10-10T02:00:10.1132723Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1133475Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:10.1134233Z (EngineCore_DP0 pid=11644) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:10.1134706Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1135384Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:10.1136022Z (EngineCore_DP0 pid=11644) prop = get_device_properties(device) 2025-10-10T02:00:10.1136447Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:10.1137102Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:10.1137768Z (EngineCore_DP0 pid=11644) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:10.1138168Z (EngineCore_DP0 pid=11644) ^^^^^^^^^^^^ 2025-10-10T02:00:10.1138795Z (EngineCore_DP0 pid=11644) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:10.1139368Z (EngineCore_DP0 pid=11644) raise RuntimeError( 2025-10-10T02:00:10.1140066Z (EngineCore_DP0 pid=11644) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:10.5142344Z FAILED 2025-10-10T02:00:10.5271122Z models/test_initialization.py::test_can_initialize_large_subset[LongcatFlashForCausalLM] Fork a new process to run a test 11648 2025-10-10T02:00:10.5283124Z Fork a new process to run a test 0 2025-10-10T02:00:10.5562294Z INFO 10-10 02:00:10 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LongcatFlashForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'meituan-longcat/LongCat-Flash-Chat'} 2025-10-10T02:00:11.3592129Z INFO 10-10 02:00:11 [model.py:551] Resolved architecture: LongcatFlashForCausalLM 2025-10-10T02:00:11.3592658Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:00:11.6167907Z 2025-10-10T02:00:12.8801786Z Parse safetensors files: 0% 0/75 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:13.8368481Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:13.8369139Z 2025-10-10T02:00:13.8369653Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:13.8370048Z 2025-10-10T02:00:13.8370739Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:13.8371374Z 2025-10-10T02:00:13.8371771Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:13.8372149Z 2025-10-10T02:00:13.8372508Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8372858Z 2025-10-10T02:00:13.8373484Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:13.8374049Z 2025-10-10T02:00:13.8374480Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:13.8375043Z 2025-10-10T02:00:13.8375901Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:13.8376408Z 2025-10-10T02:00:13.8376760Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:13.8377083Z 2025-10-10T02:00:13.8377367Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8377665Z 2025-10-10T02:00:13.8378178Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:13.8379261Z 2025-10-10T02:00:13.8379782Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] self._init_executor() 2025-10-10T02:00:13.8380277Z 2025-10-10T02:00:13.8381189Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:13.8382219Z 2025-10-10T02:00:13.8382626Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:13.8382963Z 2025-10-10T02:00:13.8383632Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:13.8384146Z 2025-10-10T02:00:13.8384507Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:13.8384848Z 2025-10-10T02:00:13.8385153Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8385438Z 2025-10-10T02:00:13.8386040Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:13.8386513Z 2025-10-10T02:00:13.8386789Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:13.8387069Z 2025-10-10T02:00:13.8387322Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8387584Z 2025-10-10T02:00:13.8388098Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:13.8388582Z 2025-10-10T02:00:13.8388869Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:13.8389159Z 2025-10-10T02:00:13.8389435Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8389707Z 2025-10-10T02:00:13.8390232Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:13.8390715Z 2025-10-10T02:00:13.8391022Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:13.8391319Z 2025-10-10T02:00:13.8391590Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8391872Z 2025-10-10T02:00:13.8392307Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:13.8392722Z 2025-10-10T02:00:13.8393063Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:13.8393401Z 2025-10-10T02:00:13.8393689Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8393986Z 2025-10-10T02:00:13.8394336Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:13.8394671Z 2025-10-10T02:00:13.8395032Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:13.8395425Z 2025-10-10T02:00:13.8395803Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:13.8396519Z 2025-10-10T02:00:13.8396972Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:13.8397364Z 2025-10-10T02:00:13.8397745Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:13.8398104Z 2025-10-10T02:00:13.8398480Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:13.8398847Z 2025-10-10T02:00:13.8399476Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:13.8399949Z 2025-10-10T02:00:13.8400298Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:13.8400635Z 2025-10-10T02:00:13.8401212Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:13.8401697Z 2025-10-10T02:00:13.8402082Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:13.8402439Z 2025-10-10T02:00:13.8403002Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:13.8403516Z 2025-10-10T02:00:13.8403816Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:13.8404116Z 2025-10-10T02:00:13.8404750Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:13.8405323Z 2025-10-10T02:00:13.8405690Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:13.8406058Z 2025-10-10T02:00:13.8406317Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8406586Z 2025-10-10T02:00:13.8407147Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:13.8407673Z 2025-10-10T02:00:13.8407971Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:13.8408351Z 2025-10-10T02:00:13.8408752Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8409032Z 2025-10-10T02:00:13.8409651Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:13.8410203Z 2025-10-10T02:00:13.8410488Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:13.8410774Z 2025-10-10T02:00:13.8411037Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8411404Z 2025-10-10T02:00:13.8411990Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:13.8412568Z 2025-10-10T02:00:13.8412950Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:13.8413272Z 2025-10-10T02:00:13.8413549Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8413834Z 2025-10-10T02:00:13.8414375Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:13.8414868Z 2025-10-10T02:00:13.8415153Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:13.8415448Z 2025-10-10T02:00:13.8415723Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8416009Z 2025-10-10T02:00:13.8416541Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:13.8417066Z 2025-10-10T02:00:13.8417393Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:13.8417702Z 2025-10-10T02:00:13.8417928Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:13.8418179Z 2025-10-10T02:00:13.8418661Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:13.8419125Z 2025-10-10T02:00:13.8419376Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] raise RuntimeError( 2025-10-10T02:00:13.8419644Z 2025-10-10T02:00:13.8420207Z (EngineCore_DP0 pid=11676) ERROR 10-10 02:00:13 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:13.8420907Z (EngineCore_DP0 pid=11676) Process EngineCore_DP0: 2025-10-10T02:00:13.8421303Z (EngineCore_DP0 pid=11676) Traceback (most recent call last): 2025-10-10T02:00:13.8421931Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:13.8422444Z (EngineCore_DP0 pid=11676) self.run() 2025-10-10T02:00:13.8422964Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:13.8423532Z (EngineCore_DP0 pid=11676) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:13.8424201Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:13.8424752Z (EngineCore_DP0 pid=11676) raise e 2025-10-10T02:00:13.8425351Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:13.8425977Z (EngineCore_DP0 pid=11676) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:13.8426440Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8427063Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:13.8427757Z (EngineCore_DP0 pid=11676) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:13.8428468Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:13.8429127Z (EngineCore_DP0 pid=11676) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:13.8429586Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8430239Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:13.8430823Z (EngineCore_DP0 pid=11676) self._init_executor() 2025-10-10T02:00:13.8431501Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:13.8432194Z (EngineCore_DP0 pid=11676) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:13.8432915Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:13.8433634Z (EngineCore_DP0 pid=11676) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:13.8434181Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8434833Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:13.8435423Z (EngineCore_DP0 pid=11676) return func(*args, **kwargs) 2025-10-10T02:00:13.8435807Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8436452Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:13.8437086Z (EngineCore_DP0 pid=11676) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:13.8437518Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8438195Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:13.8438860Z (EngineCore_DP0 pid=11676) module = importlib.import_module(module_name) 2025-10-10T02:00:13.8439413Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8439999Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:13.8440632Z (EngineCore_DP0 pid=11676) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:13.8441128Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8441626Z (EngineCore_DP0 pid=11676) File "", line 1387, in _gcd_import 2025-10-10T02:00:13.8442190Z (EngineCore_DP0 pid=11676) File "", line 1360, in _find_and_load 2025-10-10T02:00:13.8442781Z (EngineCore_DP0 pid=11676) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:13.8443365Z (EngineCore_DP0 pid=11676) File "", line 935, in _load_unlocked 2025-10-10T02:00:13.8443942Z (EngineCore_DP0 pid=11676) File "", line 999, in exec_module 2025-10-10T02:00:13.8444539Z (EngineCore_DP0 pid=11676) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:13.8445332Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:13.8446068Z (EngineCore_DP0 pid=11676) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:13.8446838Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:13.8447564Z (EngineCore_DP0 pid=11676) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:13.8448346Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:13.8449009Z (EngineCore_DP0 pid=11676) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:13.8449801Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:13.8450617Z (EngineCore_DP0 pid=11676) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:13.8451116Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8451851Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:13.8452547Z (EngineCore_DP0 pid=11676) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:13.8452992Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8453743Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:13.8454454Z (EngineCore_DP0 pid=11676) return _is_fa2_supported(device)[0] 2025-10-10T02:00:13.8454868Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8455583Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:13.8456310Z (EngineCore_DP0 pid=11676) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:13.8456772Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8457447Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:13.8458081Z (EngineCore_DP0 pid=11676) prop = get_device_properties(device) 2025-10-10T02:00:13.8458526Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:13.8459186Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:13.8459854Z (EngineCore_DP0 pid=11676) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:13.8460255Z (EngineCore_DP0 pid=11676) ^^^^^^^^^^^^ 2025-10-10T02:00:13.8460828Z (EngineCore_DP0 pid=11676) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:13.8461393Z (EngineCore_DP0 pid=11676) raise RuntimeError( 2025-10-10T02:00:13.8462090Z (EngineCore_DP0 pid=11676) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:14.2506324Z FAILED 2025-10-10T02:00:14.2635370Z models/test_initialization.py::test_can_initialize_large_subset[Gemma2ForCausalLM] Fork a new process to run a test 11680 2025-10-10T02:00:14.2647040Z Fork a new process to run a test 0 2025-10-10T02:00:14.2920082Z INFO 10-10 02:00:14 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/gemma-2-9b'} 2025-10-10T02:00:14.5395722Z 2025-10-10T02:00:14.5397243Z config.json: 0% 0.00/856 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:17.6775271Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:17.6775781Z 2025-10-10T02:00:17.6776249Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:17.6776706Z 2025-10-10T02:00:17.6777408Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:17.6778006Z 2025-10-10T02:00:17.6778410Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:17.6778784Z 2025-10-10T02:00:17.6779362Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6779670Z 2025-10-10T02:00:17.6780183Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:17.6780703Z 2025-10-10T02:00:17.6781143Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:17.6781605Z 2025-10-10T02:00:17.6782101Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:17.6782666Z 2025-10-10T02:00:17.6783013Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:17.6783332Z 2025-10-10T02:00:17.6783623Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6783896Z 2025-10-10T02:00:17.6784410Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:17.6784939Z 2025-10-10T02:00:17.6785279Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] self._init_executor() 2025-10-10T02:00:17.6785712Z 2025-10-10T02:00:17.6786478Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:17.6787111Z 2025-10-10T02:00:17.6787751Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:17.6788182Z 2025-10-10T02:00:17.6789018Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:17.6789570Z 2025-10-10T02:00:17.6790039Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:17.6790408Z 2025-10-10T02:00:17.6790982Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6791304Z 2025-10-10T02:00:17.6791943Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:17.6792510Z 2025-10-10T02:00:17.6792816Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:17.6793108Z 2025-10-10T02:00:17.6793381Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6793650Z 2025-10-10T02:00:17.6794182Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:17.6794677Z 2025-10-10T02:00:17.6794979Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:17.6795283Z 2025-10-10T02:00:17.6795554Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6795838Z 2025-10-10T02:00:17.6796725Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:17.6797249Z 2025-10-10T02:00:17.6797583Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:17.6797884Z 2025-10-10T02:00:17.6798167Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6798485Z 2025-10-10T02:00:17.6798930Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:17.6799475Z 2025-10-10T02:00:17.6799833Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:17.6800176Z 2025-10-10T02:00:17.6800477Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6800761Z 2025-10-10T02:00:17.6801110Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:17.6801450Z 2025-10-10T02:00:17.6801804Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:17.6802141Z 2025-10-10T02:00:17.6802525Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:17.6802885Z 2025-10-10T02:00:17.6803226Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:17.6803566Z 2025-10-10T02:00:17.6803931Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:17.6804302Z 2025-10-10T02:00:17.6804676Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:17.6805040Z 2025-10-10T02:00:17.6805557Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:17.6806105Z 2025-10-10T02:00:17.6806460Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:17.6806792Z 2025-10-10T02:00:17.6807373Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:17.6807916Z 2025-10-10T02:00:17.6808303Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:17.6808670Z 2025-10-10T02:00:17.6809223Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:17.6809725Z 2025-10-10T02:00:17.6810030Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:17.6810336Z 2025-10-10T02:00:17.6810960Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:17.6811537Z 2025-10-10T02:00:17.6811902Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:17.6812304Z 2025-10-10T02:00:17.6812571Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6812839Z 2025-10-10T02:00:17.6813406Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:17.6813932Z 2025-10-10T02:00:17.6814234Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:17.6814531Z 2025-10-10T02:00:17.6814809Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6815081Z 2025-10-10T02:00:17.6815693Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:17.6816257Z 2025-10-10T02:00:17.6816546Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:17.6816835Z 2025-10-10T02:00:17.6817107Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6817385Z 2025-10-10T02:00:17.6817989Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:17.6818528Z 2025-10-10T02:00:17.6818847Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:17.6819168Z 2025-10-10T02:00:17.6819440Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6819716Z 2025-10-10T02:00:17.6820239Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:17.6820723Z 2025-10-10T02:00:17.6821013Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:17.6821298Z 2025-10-10T02:00:17.6821637Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6821913Z 2025-10-10T02:00:17.6822489Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:17.6823011Z 2025-10-10T02:00:17.6823335Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:17.6823654Z 2025-10-10T02:00:17.6823884Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:17.6824134Z 2025-10-10T02:00:17.6824618Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:17.6825082Z 2025-10-10T02:00:17.6825336Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] raise RuntimeError( 2025-10-10T02:00:17.6825601Z 2025-10-10T02:00:17.6826183Z (EngineCore_DP0 pid=11711) ERROR 10-10 02:00:17 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:17.6826884Z (EngineCore_DP0 pid=11711) Process EngineCore_DP0: 2025-10-10T02:00:17.6827324Z (EngineCore_DP0 pid=11711) Traceback (most recent call last): 2025-10-10T02:00:17.6827940Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:17.6828466Z (EngineCore_DP0 pid=11711) self.run() 2025-10-10T02:00:17.6828998Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:17.6829565Z (EngineCore_DP0 pid=11711) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:17.6830243Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:17.6830799Z (EngineCore_DP0 pid=11711) raise e 2025-10-10T02:00:17.6831401Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:17.6832061Z (EngineCore_DP0 pid=11711) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:17.6832518Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6833144Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:17.6833782Z (EngineCore_DP0 pid=11711) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:17.6834452Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:17.6835075Z (EngineCore_DP0 pid=11711) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:17.6835528Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6836189Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:17.6836788Z (EngineCore_DP0 pid=11711) self._init_executor() 2025-10-10T02:00:17.6837464Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:17.6838158Z (EngineCore_DP0 pid=11711) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:17.6838956Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:17.6839810Z (EngineCore_DP0 pid=11711) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:17.6840348Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6840990Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:17.6841581Z (EngineCore_DP0 pid=11711) return func(*args, **kwargs) 2025-10-10T02:00:17.6841979Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6842620Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:17.6843261Z (EngineCore_DP0 pid=11711) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:17.6843693Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6844369Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:17.6845092Z (EngineCore_DP0 pid=11711) module = importlib.import_module(module_name) 2025-10-10T02:00:17.6845546Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6846135Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:17.6846755Z (EngineCore_DP0 pid=11711) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:17.6847250Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6847749Z (EngineCore_DP0 pid=11711) File "", line 1387, in _gcd_import 2025-10-10T02:00:17.6848309Z (EngineCore_DP0 pid=11711) File "", line 1360, in _find_and_load 2025-10-10T02:00:17.6848907Z (EngineCore_DP0 pid=11711) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:17.6849496Z (EngineCore_DP0 pid=11711) File "", line 935, in _load_unlocked 2025-10-10T02:00:17.6850078Z (EngineCore_DP0 pid=11711) File "", line 999, in exec_module 2025-10-10T02:00:17.6850682Z (EngineCore_DP0 pid=11711) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:17.6851416Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:17.6852099Z (EngineCore_DP0 pid=11711) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:17.6852826Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:17.6853554Z (EngineCore_DP0 pid=11711) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:17.6854336Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:17.6855012Z (EngineCore_DP0 pid=11711) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:17.6855848Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:17.6856724Z (EngineCore_DP0 pid=11711) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:17.6857204Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6857940Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:17.6858677Z (EngineCore_DP0 pid=11711) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:17.6859130Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6859897Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:17.6860607Z (EngineCore_DP0 pid=11711) return _is_fa2_supported(device)[0] 2025-10-10T02:00:17.6861040Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6861766Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:17.6862502Z (EngineCore_DP0 pid=11711) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:17.6863024Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6863727Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:17.6871250Z (EngineCore_DP0 pid=11711) prop = get_device_properties(device) 2025-10-10T02:00:17.6871833Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:17.6872581Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:17.6873291Z (EngineCore_DP0 pid=11711) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:17.6873704Z (EngineCore_DP0 pid=11711) ^^^^^^^^^^^^ 2025-10-10T02:00:17.6874300Z (EngineCore_DP0 pid=11711) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:17.6874878Z (EngineCore_DP0 pid=11711) raise RuntimeError( 2025-10-10T02:00:17.6875584Z (EngineCore_DP0 pid=11711) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:18.0961892Z FAILED 2025-10-10T02:00:18.1090778Z models/test_initialization.py::test_can_initialize_large_subset[Qwen3VLForConditionalGeneration] Fork a new process to run a test 11715 2025-10-10T02:00:18.1101216Z Fork a new process to run a test 0 2025-10-10T02:00:18.1102472Z Model is not available online 2025-10-10T02:00:18.4110274Z PASSED 2025-10-10T02:00:18.4239012Z models/test_initialization.py::test_can_initialize_large_subset[LlamaModel] Fork a new process to run a test 11716 2025-10-10T02:00:18.4250463Z Fork a new process to run a test 0 2025-10-10T02:00:18.4251865Z Model is not available online 2025-10-10T02:00:18.7319449Z PASSED 2025-10-10T02:00:18.7448584Z models/test_initialization.py::test_can_initialize_large_subset[Phi4MultimodalForCausalLM] Fork a new process to run a test 11717 2025-10-10T02:00:18.7459772Z Fork a new process to run a test 0 2025-10-10T02:00:18.7732725Z INFO 10-10 02:00:18 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'revision': 'refs/pr/70', 'hf_overrides': functools.partial(, model_arch='Phi4MultimodalForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'microsoft/Phi-4-multimodal-instruct'} 2025-10-10T02:00:18.9442399Z 2025-10-10T02:00:18.9444580Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:18.9444983Z config.json: 3.71kB [00:00, 19.0MB/s] 2025-10-10T02:00:18.9933784Z INFO 10-10 02:00:18 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T02:00:19.5703059Z 2025-10-10T02:00:19.5704282Z preprocessor_config.json: 0% 0.00/705 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:28.8521989Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:28.8522755Z 2025-10-10T02:00:28.8523200Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:28.8523585Z 2025-10-10T02:00:28.8524271Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:28.8524914Z 2025-10-10T02:00:28.8525310Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:28.8525688Z 2025-10-10T02:00:28.8526058Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8526415Z 2025-10-10T02:00:28.8527306Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:28.8528004Z 2025-10-10T02:00:28.8528844Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:28.8529218Z 2025-10-10T02:00:28.8529734Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:28.8530215Z 2025-10-10T02:00:28.8530539Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:28.8530856Z 2025-10-10T02:00:28.8531142Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8531533Z 2025-10-10T02:00:28.8532542Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:28.8533433Z 2025-10-10T02:00:28.8533892Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] self._init_executor() 2025-10-10T02:00:28.8534281Z 2025-10-10T02:00:28.8534930Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:28.8535476Z 2025-10-10T02:00:28.8535838Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:28.8536174Z 2025-10-10T02:00:28.8536732Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:28.8537249Z 2025-10-10T02:00:28.8537605Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:28.8537951Z 2025-10-10T02:00:28.8538241Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8538695Z 2025-10-10T02:00:28.8539207Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:28.8539672Z 2025-10-10T02:00:28.8540035Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:28.8540404Z 2025-10-10T02:00:28.8540673Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8540947Z 2025-10-10T02:00:28.8541483Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:28.8541963Z 2025-10-10T02:00:28.8542259Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:28.8542546Z 2025-10-10T02:00:28.8542826Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8543097Z 2025-10-10T02:00:28.8543660Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:28.8544384Z 2025-10-10T02:00:28.8544787Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:28.8545118Z 2025-10-10T02:00:28.8545402Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8545681Z 2025-10-10T02:00:28.8546134Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:28.8546562Z 2025-10-10T02:00:28.8546905Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:28.8547235Z 2025-10-10T02:00:28.8547532Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8547813Z 2025-10-10T02:00:28.8548169Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:28.8548499Z 2025-10-10T02:00:28.8548854Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:28.8549189Z 2025-10-10T02:00:28.8549564Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:28.8549929Z 2025-10-10T02:00:28.8550274Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:28.8550611Z 2025-10-10T02:00:28.8550972Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:28.8551330Z 2025-10-10T02:00:28.8551705Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:28.8552072Z 2025-10-10T02:00:28.8552569Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:28.8553033Z 2025-10-10T02:00:28.8553392Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:28.8553792Z 2025-10-10T02:00:28.8554325Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:28.8554809Z 2025-10-10T02:00:28.8555233Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:28.8555635Z 2025-10-10T02:00:28.8556196Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:28.8556706Z 2025-10-10T02:00:28.8557001Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:28.8557304Z 2025-10-10T02:00:28.8557940Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:28.8558520Z 2025-10-10T02:00:28.8558898Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:28.8559415Z 2025-10-10T02:00:28.8559678Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8560015Z 2025-10-10T02:00:28.8560594Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:28.8561119Z 2025-10-10T02:00:28.8561420Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:28.8561724Z 2025-10-10T02:00:28.8561998Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8562279Z 2025-10-10T02:00:28.8562878Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:28.8563430Z 2025-10-10T02:00:28.8563715Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:28.8564006Z 2025-10-10T02:00:28.8564270Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8564536Z 2025-10-10T02:00:28.8565119Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:28.8565645Z 2025-10-10T02:00:28.8565968Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:28.8566279Z 2025-10-10T02:00:28.8566554Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8566835Z 2025-10-10T02:00:28.8567356Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:28.8567847Z 2025-10-10T02:00:28.8568127Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:28.8568424Z 2025-10-10T02:00:28.8568695Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8568971Z 2025-10-10T02:00:28.8569567Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:28.8570050Z 2025-10-10T02:00:28.8570440Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:28.8570797Z 2025-10-10T02:00:28.8571039Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:28.8571284Z 2025-10-10T02:00:28.8571766Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:28.8572220Z 2025-10-10T02:00:28.8572467Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] raise RuntimeError( 2025-10-10T02:00:28.8572733Z 2025-10-10T02:00:28.8573300Z (EngineCore_DP0 pid=11795) ERROR 10-10 02:00:28 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:28.8574021Z (EngineCore_DP0 pid=11795) Process EngineCore_DP0: 2025-10-10T02:00:28.8574425Z (EngineCore_DP0 pid=11795) Traceback (most recent call last): 2025-10-10T02:00:28.8575091Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:28.8575617Z (EngineCore_DP0 pid=11795) self.run() 2025-10-10T02:00:28.8576145Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:28.8576706Z (EngineCore_DP0 pid=11795) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:28.8577386Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:28.8577941Z (EngineCore_DP0 pid=11795) raise e 2025-10-10T02:00:28.8578543Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:28.8579186Z (EngineCore_DP0 pid=11795) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:28.8579645Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8580269Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:28.8580915Z (EngineCore_DP0 pid=11795) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:28.8581582Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:28.8582207Z (EngineCore_DP0 pid=11795) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:28.8582658Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8583316Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:28.8583910Z (EngineCore_DP0 pid=11795) self._init_executor() 2025-10-10T02:00:28.8584583Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:28.8585273Z (EngineCore_DP0 pid=11795) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:28.8585998Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:28.8586788Z (EngineCore_DP0 pid=11795) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:28.8587290Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8587973Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:28.8588614Z (EngineCore_DP0 pid=11795) return func(*args, **kwargs) 2025-10-10T02:00:28.8589016Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8589656Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:28.8590295Z (EngineCore_DP0 pid=11795) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:28.8590731Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8591402Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:28.8592067Z (EngineCore_DP0 pid=11795) module = importlib.import_module(module_name) 2025-10-10T02:00:28.8592562Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8593155Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:28.8593786Z (EngineCore_DP0 pid=11795) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:28.8594296Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8594795Z (EngineCore_DP0 pid=11795) File "", line 1387, in _gcd_import 2025-10-10T02:00:28.8595349Z (EngineCore_DP0 pid=11795) File "", line 1360, in _find_and_load 2025-10-10T02:00:28.8595944Z (EngineCore_DP0 pid=11795) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:28.8596932Z (EngineCore_DP0 pid=11795) File "", line 935, in _load_unlocked 2025-10-10T02:00:28.8597518Z (EngineCore_DP0 pid=11795) File "", line 999, in exec_module 2025-10-10T02:00:28.8598126Z (EngineCore_DP0 pid=11795) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:28.8598862Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:28.8599610Z (EngineCore_DP0 pid=11795) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:28.8600337Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:28.8601067Z (EngineCore_DP0 pid=11795) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:28.8601854Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:28.8602528Z (EngineCore_DP0 pid=11795) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:28.8603316Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:28.8604134Z (EngineCore_DP0 pid=11795) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:28.8604742Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8605513Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:28.8606281Z (EngineCore_DP0 pid=11795) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:28.8606740Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8607491Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:28.8608199Z (EngineCore_DP0 pid=11795) return _is_fa2_supported(device)[0] 2025-10-10T02:00:28.8608616Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8609338Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:28.8610053Z (EngineCore_DP0 pid=11795) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:28.8610519Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8611260Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:28.8611903Z (EngineCore_DP0 pid=11795) prop = get_device_properties(device) 2025-10-10T02:00:28.8612325Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:28.8612983Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:28.8613646Z (EngineCore_DP0 pid=11795) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:28.8614043Z (EngineCore_DP0 pid=11795) ^^^^^^^^^^^^ 2025-10-10T02:00:28.8614629Z (EngineCore_DP0 pid=11795) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:28.8615206Z (EngineCore_DP0 pid=11795) raise RuntimeError( 2025-10-10T02:00:28.8615903Z (EngineCore_DP0 pid=11795) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:29.2827533Z FAILED 2025-10-10T02:00:29.2956830Z models/test_initialization.py::test_can_initialize_large_subset[Glm4vMoeForConditionalGeneration] Fork a new process to run a test 11799 2025-10-10T02:00:29.2968079Z Fork a new process to run a test 0 2025-10-10T02:00:29.3237793Z INFO 10-10 02:00:29 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Glm4vMoeForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'zai-org/GLM-4.5V'} 2025-10-10T02:00:29.4070655Z 2025-10-10T02:00:29.4072710Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:29.4073087Z config.json: 1.86kB [00:00, 10.7MB/s] 2025-10-10T02:00:29.5333713Z 2025-10-10T02:00:29.5334848Z preprocessor_config.json: 0% 0.00/364 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:38.5218836Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:38.5219551Z 2025-10-10T02:00:38.5220226Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:38.5220877Z 2025-10-10T02:00:38.5221817Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:38.5222433Z 2025-10-10T02:00:38.5222827Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:38.5223211Z 2025-10-10T02:00:38.5223558Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5224167Z 2025-10-10T02:00:38.5224870Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:38.5225563Z 2025-10-10T02:00:38.5225996Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:38.5226405Z 2025-10-10T02:00:38.5227015Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:38.5227568Z 2025-10-10T02:00:38.5227971Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:38.5228277Z 2025-10-10T02:00:38.5228564Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5228839Z 2025-10-10T02:00:38.5229351Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:38.5229824Z 2025-10-10T02:00:38.5230854Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] self._init_executor() 2025-10-10T02:00:38.5231160Z 2025-10-10T02:00:38.5231856Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:38.5232780Z 2025-10-10T02:00:38.5233353Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:38.5233907Z 2025-10-10T02:00:38.5234491Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:38.5235001Z 2025-10-10T02:00:38.5235352Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:38.5235693Z 2025-10-10T02:00:38.5235989Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5236272Z 2025-10-10T02:00:38.5236763Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:38.5237215Z 2025-10-10T02:00:38.5237487Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:38.5237769Z 2025-10-10T02:00:38.5238032Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5238293Z 2025-10-10T02:00:38.5238807Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:38.5239434Z 2025-10-10T02:00:38.5239733Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:38.5240031Z 2025-10-10T02:00:38.5240300Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5240580Z 2025-10-10T02:00:38.5241107Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:38.5241680Z 2025-10-10T02:00:38.5241996Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:38.5242298Z 2025-10-10T02:00:38.5242622Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5242952Z 2025-10-10T02:00:38.5243406Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:38.5243818Z 2025-10-10T02:00:38.5244164Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:38.5244489Z 2025-10-10T02:00:38.5244776Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5245063Z 2025-10-10T02:00:38.5245402Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:38.5245735Z 2025-10-10T02:00:38.5246087Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:38.5246434Z 2025-10-10T02:00:38.5246855Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:38.5247224Z 2025-10-10T02:00:38.5247585Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:38.5247915Z 2025-10-10T02:00:38.5248277Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:38.5248627Z 2025-10-10T02:00:38.5249003Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:38.5249359Z 2025-10-10T02:00:38.5249868Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:38.5250333Z 2025-10-10T02:00:38.5250691Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:38.5251052Z 2025-10-10T02:00:38.5251581Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:38.5252064Z 2025-10-10T02:00:38.5252435Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:38.5252802Z 2025-10-10T02:00:38.5253356Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:38.5253870Z 2025-10-10T02:00:38.5254161Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:38.5254456Z 2025-10-10T02:00:38.5255085Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:38.5255656Z 2025-10-10T02:00:38.5256023Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:38.5256423Z 2025-10-10T02:00:38.5256687Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5256947Z 2025-10-10T02:00:38.5257550Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:38.5258125Z 2025-10-10T02:00:38.5258424Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:38.5258723Z 2025-10-10T02:00:38.5258994Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5259270Z 2025-10-10T02:00:38.5259880Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:38.5260439Z 2025-10-10T02:00:38.5260722Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:38.5261003Z 2025-10-10T02:00:38.5261272Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5261542Z 2025-10-10T02:00:38.5262169Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:38.5262695Z 2025-10-10T02:00:38.5263024Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:38.5263336Z 2025-10-10T02:00:38.5263616Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5263907Z 2025-10-10T02:00:38.5264433Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:38.5264923Z 2025-10-10T02:00:38.5265207Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:38.5265502Z 2025-10-10T02:00:38.5265773Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5266047Z 2025-10-10T02:00:38.5266573Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:38.5267056Z 2025-10-10T02:00:38.5267375Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:38.5267692Z 2025-10-10T02:00:38.5267925Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:38.5268168Z 2025-10-10T02:00:38.5268651Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:38.5269108Z 2025-10-10T02:00:38.5269369Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] raise RuntimeError( 2025-10-10T02:00:38.5269643Z 2025-10-10T02:00:38.5270211Z (EngineCore_DP0 pid=11882) ERROR 10-10 02:00:38 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:38.5270933Z (EngineCore_DP0 pid=11882) Process EngineCore_DP0: 2025-10-10T02:00:38.5271338Z (EngineCore_DP0 pid=11882) Traceback (most recent call last): 2025-10-10T02:00:38.5272003Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:38.5272514Z (EngineCore_DP0 pid=11882) self.run() 2025-10-10T02:00:38.5273106Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:38.5273712Z (EngineCore_DP0 pid=11882) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:38.5274397Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:38.5274943Z (EngineCore_DP0 pid=11882) raise e 2025-10-10T02:00:38.5275530Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:38.5276180Z (EngineCore_DP0 pid=11882) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:38.5276617Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5277249Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:38.5277893Z (EngineCore_DP0 pid=11882) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:38.5278594Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:38.5279314Z (EngineCore_DP0 pid=11882) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:38.5279770Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5280419Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:38.5281011Z (EngineCore_DP0 pid=11882) self._init_executor() 2025-10-10T02:00:38.5281716Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:38.5282437Z (EngineCore_DP0 pid=11882) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:38.5283169Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:38.5283879Z (EngineCore_DP0 pid=11882) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:38.5284381Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5285023Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:38.5285613Z (EngineCore_DP0 pid=11882) return func(*args, **kwargs) 2025-10-10T02:00:38.5286007Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5286642Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:38.5287283Z (EngineCore_DP0 pid=11882) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:38.5287714Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5288385Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:38.5289042Z (EngineCore_DP0 pid=11882) module = importlib.import_module(module_name) 2025-10-10T02:00:38.5289538Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5290124Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:38.5290793Z (EngineCore_DP0 pid=11882) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:38.5291327Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5291824Z (EngineCore_DP0 pid=11882) File "", line 1387, in _gcd_import 2025-10-10T02:00:38.5292388Z (EngineCore_DP0 pid=11882) File "", line 1360, in _find_and_load 2025-10-10T02:00:38.5292983Z (EngineCore_DP0 pid=11882) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:38.5293569Z (EngineCore_DP0 pid=11882) File "", line 935, in _load_unlocked 2025-10-10T02:00:38.5294144Z (EngineCore_DP0 pid=11882) File "", line 999, in exec_module 2025-10-10T02:00:38.5294751Z (EngineCore_DP0 pid=11882) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:38.5295525Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:38.5296415Z (EngineCore_DP0 pid=11882) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:38.5297148Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:38.5297872Z (EngineCore_DP0 pid=11882) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:38.5298661Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:38.5299334Z (EngineCore_DP0 pid=11882) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:38.5300120Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:38.5300937Z (EngineCore_DP0 pid=11882) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:38.5301418Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5302110Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:38.5302793Z (EngineCore_DP0 pid=11882) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:38.5303232Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5303982Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:38.5304688Z (EngineCore_DP0 pid=11882) return _is_fa2_supported(device)[0] 2025-10-10T02:00:38.5305105Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5305824Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:38.5306534Z (EngineCore_DP0 pid=11882) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:38.5306992Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5307773Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:38.5308473Z (EngineCore_DP0 pid=11882) prop = get_device_properties(device) 2025-10-10T02:00:38.5308967Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:38.5309684Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:38.5310373Z (EngineCore_DP0 pid=11882) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:38.5310770Z (EngineCore_DP0 pid=11882) ^^^^^^^^^^^^ 2025-10-10T02:00:38.5311354Z (EngineCore_DP0 pid=11882) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:38.5311925Z (EngineCore_DP0 pid=11882) raise RuntimeError( 2025-10-10T02:00:38.5312623Z (EngineCore_DP0 pid=11882) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:38.9575579Z FAILED 2025-10-10T02:00:38.9706329Z models/test_initialization.py::test_can_initialize_large_subset[BertModel] Fork a new process to run a test 11886 2025-10-10T02:00:38.9717843Z Fork a new process to run a test 0 2025-10-10T02:00:38.9998686Z INFO 10-10 02:00:38 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='BertModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/bge-base-en-v1.5'} 2025-10-10T02:00:39.0858061Z 2025-10-10T02:00:39.0859551Z config.json: 0% 0.00/777 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:40.1092711Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:40.1093278Z 2025-10-10T02:00:40.1093723Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:40.1094119Z 2025-10-10T02:00:40.1094823Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:40.1095431Z 2025-10-10T02:00:40.1095830Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:40.1096571Z 2025-10-10T02:00:40.1096956Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1097317Z 2025-10-10T02:00:40.1098187Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:40.1098756Z 2025-10-10T02:00:40.1099282Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:40.1099828Z 2025-10-10T02:00:40.1100376Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:40.1100839Z 2025-10-10T02:00:40.1101182Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:40.1101502Z 2025-10-10T02:00:40.1101796Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1102081Z 2025-10-10T02:00:40.1102610Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:40.1103195Z 2025-10-10T02:00:40.1103556Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] self._init_executor() 2025-10-10T02:00:40.1103904Z 2025-10-10T02:00:40.1104781Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:40.1105456Z 2025-10-10T02:00:40.1106096Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:40.1106685Z 2025-10-10T02:00:40.1107655Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:40.1108341Z 2025-10-10T02:00:40.1108702Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:40.1109091Z 2025-10-10T02:00:40.1109398Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1109704Z 2025-10-10T02:00:40.1110208Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:40.1110675Z 2025-10-10T02:00:40.1110948Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:40.1111229Z 2025-10-10T02:00:40.1111486Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1111759Z 2025-10-10T02:00:40.1112292Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:40.1112770Z 2025-10-10T02:00:40.1113071Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:40.1113365Z 2025-10-10T02:00:40.1113647Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1113922Z 2025-10-10T02:00:40.1114461Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:40.1114978Z 2025-10-10T02:00:40.1115289Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:40.1115686Z 2025-10-10T02:00:40.1115971Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1116257Z 2025-10-10T02:00:40.1116752Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:40.1117245Z 2025-10-10T02:00:40.1117596Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:40.1117929Z 2025-10-10T02:00:40.1118228Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1118519Z 2025-10-10T02:00:40.1118877Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:40.1119338Z 2025-10-10T02:00:40.1119703Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:40.1120048Z 2025-10-10T02:00:40.1120431Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:40.1120815Z 2025-10-10T02:00:40.1121217Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:40.1121580Z 2025-10-10T02:00:40.1121950Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:40.1122312Z 2025-10-10T02:00:40.1122689Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:40.1123064Z 2025-10-10T02:00:40.1123569Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:40.1124040Z 2025-10-10T02:00:40.1124400Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:40.1124746Z 2025-10-10T02:00:40.1125289Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:40.1125774Z 2025-10-10T02:00:40.1126156Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:40.1126518Z 2025-10-10T02:00:40.1127074Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:40.1127598Z 2025-10-10T02:00:40.1127896Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:40.1128202Z 2025-10-10T02:00:40.1128838Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:40.1129424Z 2025-10-10T02:00:40.1129795Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:40.1130157Z 2025-10-10T02:00:40.1130415Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1130728Z 2025-10-10T02:00:40.1131301Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:40.1131823Z 2025-10-10T02:00:40.1132167Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:40.1132543Z 2025-10-10T02:00:40.1132830Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1133107Z 2025-10-10T02:00:40.1133716Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:40.1134270Z 2025-10-10T02:00:40.1134555Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:40.1134849Z 2025-10-10T02:00:40.1135115Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1135403Z 2025-10-10T02:00:40.1135989Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:40.1136529Z 2025-10-10T02:00:40.1136893Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:40.1137221Z 2025-10-10T02:00:40.1137500Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1137775Z 2025-10-10T02:00:40.1138308Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:40.1138803Z 2025-10-10T02:00:40.1139097Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:40.1139388Z 2025-10-10T02:00:40.1139668Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1139944Z 2025-10-10T02:00:40.1140468Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:40.1140958Z 2025-10-10T02:00:40.1141272Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:40.1141584Z 2025-10-10T02:00:40.1141815Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:40.1142066Z 2025-10-10T02:00:40.1142548Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:40.1143015Z 2025-10-10T02:00:40.1143270Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] raise RuntimeError( 2025-10-10T02:00:40.1143536Z 2025-10-10T02:00:40.1144118Z (EngineCore_DP0 pid=11894) ERROR 10-10 02:00:40 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:40.1144830Z (EngineCore_DP0 pid=11894) Process EngineCore_DP0: 2025-10-10T02:00:40.1145235Z (EngineCore_DP0 pid=11894) Traceback (most recent call last): 2025-10-10T02:00:40.1145855Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:40.1146433Z (EngineCore_DP0 pid=11894) self.run() 2025-10-10T02:00:40.1146977Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:40.1147591Z (EngineCore_DP0 pid=11894) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:40.1148305Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:40.1148874Z (EngineCore_DP0 pid=11894) raise e 2025-10-10T02:00:40.1149476Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:40.1150127Z (EngineCore_DP0 pid=11894) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:40.1150583Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1151212Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:40.1151862Z (EngineCore_DP0 pid=11894) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:40.1152572Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:40.1153206Z (EngineCore_DP0 pid=11894) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:40.1153668Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1154337Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:40.1154935Z (EngineCore_DP0 pid=11894) self._init_executor() 2025-10-10T02:00:40.1155609Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:40.1156320Z (EngineCore_DP0 pid=11894) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:40.1157064Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:40.1157788Z (EngineCore_DP0 pid=11894) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:40.1158289Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1158942Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:40.1159629Z (EngineCore_DP0 pid=11894) return func(*args, **kwargs) 2025-10-10T02:00:40.1160031Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1160683Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:40.1161321Z (EngineCore_DP0 pid=11894) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:40.1161761Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1162439Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:40.1163105Z (EngineCore_DP0 pid=11894) module = importlib.import_module(module_name) 2025-10-10T02:00:40.1163561Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1164200Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:40.1164847Z (EngineCore_DP0 pid=11894) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:40.1165381Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1165929Z (EngineCore_DP0 pid=11894) File "", line 1387, in _gcd_import 2025-10-10T02:00:40.1166494Z (EngineCore_DP0 pid=11894) File "", line 1360, in _find_and_load 2025-10-10T02:00:40.1167085Z (EngineCore_DP0 pid=11894) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:40.1167672Z (EngineCore_DP0 pid=11894) File "", line 935, in _load_unlocked 2025-10-10T02:00:40.1168257Z (EngineCore_DP0 pid=11894) File "", line 999, in exec_module 2025-10-10T02:00:40.1168868Z (EngineCore_DP0 pid=11894) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:40.1169622Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:40.1170363Z (EngineCore_DP0 pid=11894) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:40.1171114Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:40.1171852Z (EngineCore_DP0 pid=11894) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:40.1172645Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:40.1173331Z (EngineCore_DP0 pid=11894) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:40.1174149Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:40.1174974Z (EngineCore_DP0 pid=11894) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:40.1175471Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1176172Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:40.1176869Z (EngineCore_DP0 pid=11894) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:40.1177319Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1178081Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:40.1178794Z (EngineCore_DP0 pid=11894) return _is_fa2_supported(device)[0] 2025-10-10T02:00:40.1179225Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1179946Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:40.1180680Z (EngineCore_DP0 pid=11894) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:40.1181146Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1181828Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:40.1182531Z (EngineCore_DP0 pid=11894) prop = get_device_properties(device) 2025-10-10T02:00:40.1182958Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:40.1183657Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:40.1184367Z (EngineCore_DP0 pid=11894) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:40.1184766Z (EngineCore_DP0 pid=11894) ^^^^^^^^^^^^ 2025-10-10T02:00:40.1185352Z (EngineCore_DP0 pid=11894) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:40.1185943Z (EngineCore_DP0 pid=11894) raise RuntimeError( 2025-10-10T02:00:40.1186644Z (EngineCore_DP0 pid=11894) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:40.5249885Z FAILED 2025-10-10T02:00:40.5380466Z models/test_initialization.py::test_can_initialize_large_subset[GritLM] Fork a new process to run a test 11898 2025-10-10T02:00:40.5391172Z Fork a new process to run a test 0 2025-10-10T02:00:40.5669946Z INFO 10-10 02:00:40 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GritLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'parasail-ai/GritLM-7B-vllm'} 2025-10-10T02:00:40.7127898Z 2025-10-10T02:00:40.7129145Z config.json: 0% 0.00/934 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:48.5868099Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:48.5868772Z 2025-10-10T02:00:48.5869438Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:48.5869828Z 2025-10-10T02:00:48.5870518Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:48.5871125Z 2025-10-10T02:00:48.5871522Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:48.5871904Z 2025-10-10T02:00:48.5872272Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5872621Z 2025-10-10T02:00:48.5873229Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:48.5873786Z 2025-10-10T02:00:48.5874201Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:48.5874605Z 2025-10-10T02:00:48.5875192Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:48.5875743Z 2025-10-10T02:00:48.5876135Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:48.5876763Z 2025-10-10T02:00:48.5877140Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5877421Z 2025-10-10T02:00:48.5878025Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:48.5878585Z 2025-10-10T02:00:48.5878859Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] self._init_executor() 2025-10-10T02:00:48.5879285Z 2025-10-10T02:00:48.5879841Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:48.5880338Z 2025-10-10T02:00:48.5880664Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:48.5880987Z 2025-10-10T02:00:48.5881537Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:48.5882047Z 2025-10-10T02:00:48.5882394Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:48.5882736Z 2025-10-10T02:00:48.5883126Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5883420Z 2025-10-10T02:00:48.5883915Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:48.5884371Z 2025-10-10T02:00:48.5884637Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:48.5884908Z 2025-10-10T02:00:48.5885161Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5885418Z 2025-10-10T02:00:48.5885941Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:48.5886434Z 2025-10-10T02:00:48.5886731Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:48.5887029Z 2025-10-10T02:00:48.5887298Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5887573Z 2025-10-10T02:00:48.5888099Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:48.5888591Z 2025-10-10T02:00:48.5888893Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:48.5889207Z 2025-10-10T02:00:48.5889487Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5889770Z 2025-10-10T02:00:48.5890219Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:48.5890630Z 2025-10-10T02:00:48.5891007Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:48.5891345Z 2025-10-10T02:00:48.5891633Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5891977Z 2025-10-10T02:00:48.5892325Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:48.5892662Z 2025-10-10T02:00:48.5893057Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:48.5893439Z 2025-10-10T02:00:48.5893824Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:48.5894183Z 2025-10-10T02:00:48.5894544Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:48.5894877Z 2025-10-10T02:00:48.5895242Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:48.5895594Z 2025-10-10T02:00:48.5895967Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:48.5896763Z 2025-10-10T02:00:48.5897289Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:48.5897761Z 2025-10-10T02:00:48.5898219Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:48.5898566Z 2025-10-10T02:00:48.5899091Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:48.5899576Z 2025-10-10T02:00:48.5899950Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:48.5900315Z 2025-10-10T02:00:48.5900875Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:48.5901386Z 2025-10-10T02:00:48.5901689Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:48.5901987Z 2025-10-10T02:00:48.5902614Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:48.5903182Z 2025-10-10T02:00:48.5903553Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:48.5903907Z 2025-10-10T02:00:48.5904159Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5904426Z 2025-10-10T02:00:48.5904994Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:48.5905542Z 2025-10-10T02:00:48.5905865Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:48.5906180Z 2025-10-10T02:00:48.5906455Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5906733Z 2025-10-10T02:00:48.5907348Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:48.5907986Z 2025-10-10T02:00:48.5908281Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:48.5908571Z 2025-10-10T02:00:48.5908916Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5909252Z 2025-10-10T02:00:48.5909866Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:48.5910409Z 2025-10-10T02:00:48.5910733Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:48.5911053Z 2025-10-10T02:00:48.5911327Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5911605Z 2025-10-10T02:00:48.5912132Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:48.5912624Z 2025-10-10T02:00:48.5912911Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:48.5913204Z 2025-10-10T02:00:48.5913526Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5913805Z 2025-10-10T02:00:48.5914333Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:48.5914820Z 2025-10-10T02:00:48.5915147Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:48.5915456Z 2025-10-10T02:00:48.5915687Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:48.5915928Z 2025-10-10T02:00:48.5916412Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:48.5916868Z 2025-10-10T02:00:48.5917114Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] raise RuntimeError( 2025-10-10T02:00:48.5917375Z 2025-10-10T02:00:48.5917939Z (EngineCore_DP0 pid=11977) ERROR 10-10 02:00:48 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:48.5918635Z (EngineCore_DP0 pid=11977) Process EngineCore_DP0: 2025-10-10T02:00:48.5919032Z (EngineCore_DP0 pid=11977) Traceback (most recent call last): 2025-10-10T02:00:48.5919769Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:48.5920279Z (EngineCore_DP0 pid=11977) self.run() 2025-10-10T02:00:48.5920813Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:48.5921379Z (EngineCore_DP0 pid=11977) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:48.5922050Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:48.5922596Z (EngineCore_DP0 pid=11977) raise e 2025-10-10T02:00:48.5923182Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:48.5923923Z (EngineCore_DP0 pid=11977) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:48.5924371Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5925034Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:48.5925758Z (EngineCore_DP0 pid=11977) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:48.5926426Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:48.5927045Z (EngineCore_DP0 pid=11977) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:48.5927500Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5928153Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:48.5928746Z (EngineCore_DP0 pid=11977) self._init_executor() 2025-10-10T02:00:48.5929426Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:48.5930164Z (EngineCore_DP0 pid=11977) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:48.5930906Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:48.5931630Z (EngineCore_DP0 pid=11977) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:48.5932136Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5932781Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:48.5933366Z (EngineCore_DP0 pid=11977) return func(*args, **kwargs) 2025-10-10T02:00:48.5933757Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5934406Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:48.5935042Z (EngineCore_DP0 pid=11977) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:48.5935471Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5936137Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:48.5936796Z (EngineCore_DP0 pid=11977) module = importlib.import_module(module_name) 2025-10-10T02:00:48.5937251Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5937858Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:48.5938491Z (EngineCore_DP0 pid=11977) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:48.5938982Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5939474Z (EngineCore_DP0 pid=11977) File "", line 1387, in _gcd_import 2025-10-10T02:00:48.5940025Z (EngineCore_DP0 pid=11977) File "", line 1360, in _find_and_load 2025-10-10T02:00:48.5940611Z (EngineCore_DP0 pid=11977) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:48.5941253Z (EngineCore_DP0 pid=11977) File "", line 935, in _load_unlocked 2025-10-10T02:00:48.5941834Z (EngineCore_DP0 pid=11977) File "", line 999, in exec_module 2025-10-10T02:00:48.5942472Z (EngineCore_DP0 pid=11977) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:48.5943258Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:48.5943940Z (EngineCore_DP0 pid=11977) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:48.5944663Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:48.5945386Z (EngineCore_DP0 pid=11977) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:48.5946191Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:48.5946891Z (EngineCore_DP0 pid=11977) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:48.5947742Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:48.5948587Z (EngineCore_DP0 pid=11977) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:48.5949072Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5949768Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:48.5950458Z (EngineCore_DP0 pid=11977) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:48.5950902Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5951665Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:48.5952375Z (EngineCore_DP0 pid=11977) return _is_fa2_supported(device)[0] 2025-10-10T02:00:48.5952790Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5953499Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:48.5954219Z (EngineCore_DP0 pid=11977) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:48.5954691Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5955363Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:48.5956005Z (EngineCore_DP0 pid=11977) prop = get_device_properties(device) 2025-10-10T02:00:48.5956428Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:48.5957092Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:48.5957761Z (EngineCore_DP0 pid=11977) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:48.5958161Z (EngineCore_DP0 pid=11977) ^^^^^^^^^^^^ 2025-10-10T02:00:48.5958745Z (EngineCore_DP0 pid=11977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:48.5959450Z (EngineCore_DP0 pid=11977) raise RuntimeError( 2025-10-10T02:00:48.5960191Z (EngineCore_DP0 pid=11977) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:49.0000314Z FAILED 2025-10-10T02:00:49.0129339Z models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForProcessRewardModel] Fork a new process to run a test 11981 2025-10-10T02:00:49.0141469Z Fork a new process to run a test 0 2025-10-10T02:00:49.0145562Z `transformers==4.56.2` installed, but `transformers<=4.53` is required to run this model. Reason: HF model uses remote code that is not compatible with latest Transformers 2025-10-10T02:00:49.3260233Z PASSED 2025-10-10T02:00:49.3390629Z models/test_initialization.py::test_can_initialize_large_subset[InternLM2VEForCausalLM] Fork a new process to run a test 11982 2025-10-10T02:00:49.3402940Z Fork a new process to run a test 0 2025-10-10T02:00:49.3678098Z INFO 10-10 02:00:49 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='InternLM2VEForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'OpenGVLab/Mono-InternVL-2B'} 2025-10-10T02:00:49.5119956Z 2025-10-10T02:00:49.5122931Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:49.5123270Z config.json: 3.75kB [00:00, 15.7MB/s] 2025-10-10T02:00:49.5773610Z 2025-10-10T02:00:49.5775539Z configuration_internvl_chat.py: 0.00B [00:00, ?B/s] 2025-10-10T02:00:49.5775983Z configuration_internvl_chat.py: 4.44kB [00:00, 30.3MB/s] 2025-10-10T02:00:49.6201561Z 2025-10-10T02:00:49.6203385Z configuration_intern_patch.py: 0.00B [00:00, ?B/s] 2025-10-10T02:00:49.6203798Z configuration_intern_patch.py: 4.40kB [00:00, 34.4MB/s] 2025-10-10T02:00:49.6222667Z A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/Mono-InternVL-2B: 2025-10-10T02:00:49.6223220Z - configuration_intern_patch.py 2025-10-10T02:00:49.6223860Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:00:49.6761517Z 2025-10-10T02:00:49.6763067Z configuration_internlm2.py: 0.00B [00:00, ?B/s] 2025-10-10T02:00:49.6763484Z configuration_internlm2.py: 7.00kB [00:00, 51.8MB/s] 2025-10-10T02:00:49.6785691Z A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/Mono-InternVL-2B: 2025-10-10T02:00:49.6786352Z - configuration_internlm2.py 2025-10-10T02:00:49.6786976Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:00:49.6787855Z A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/Mono-InternVL-2B: 2025-10-10T02:00:49.6788385Z - configuration_internvl_chat.py 2025-10-10T02:00:49.6788679Z - configuration_intern_patch.py 2025-10-10T02:00:49.6788943Z - configuration_internlm2.py 2025-10-10T02:00:49.6789556Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:00:49.9009360Z INFO 10-10 02:00:49 [model.py:551] Resolved architecture: InternVLChatModel 2025-10-10T02:00:49.9009826Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:00:49.9263105Z INFO 10-10 02:00:49 [model.py:1545] Using max model len 32768 2025-10-10T02:00:49.9265548Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T02:00:49.9871650Z INFO 10-10 02:00:49 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:00:50.0338925Z 2025-10-10T02:00:50.0341043Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:50.0341403Z tokenizer_config.json: 4.00kB [00:00, 14.5MB/s] 2025-10-10T02:00:50.0730233Z 2025-10-10T02:00:50.0732244Z tokenization_internlm2.py: 0.00B [00:00, ?B/s] 2025-10-10T02:00:50.0732611Z tokenization_internlm2.py: 8.79kB [00:00, 46.9MB/s] 2025-10-10T02:00:50.0784593Z A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/Mono-InternVL-2B: 2025-10-10T02:00:50.0785259Z - tokenization_internlm2.py 2025-10-10T02:00:50.0785890Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:00:50.1853230Z 2025-10-10T02:00:50.4152969Z ./tokenizer.model: 0% 0.00/1.48M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:00:51.3957456Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] EngineCore failed to start. 2025-10-10T02:00:51.3958098Z 2025-10-10T02:00:51.3958648Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] Traceback (most recent call last): 2025-10-10T02:00:51.3959024Z 2025-10-10T02:00:51.3959823Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:51.3960318Z 2025-10-10T02:00:51.3960643Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:51.3961094Z 2025-10-10T02:00:51.3961386Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3961666Z 2025-10-10T02:00:51.3962159Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:51.3962608Z 2025-10-10T02:00:51.3962945Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:51.3963349Z 2025-10-10T02:00:51.3963828Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:51.3964410Z 2025-10-10T02:00:51.3964746Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:51.3965060Z 2025-10-10T02:00:51.3965340Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3965623Z 2025-10-10T02:00:51.3966127Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:51.3966595Z 2025-10-10T02:00:51.3966852Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] self._init_executor() 2025-10-10T02:00:51.3967114Z 2025-10-10T02:00:51.3967842Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:51.3968629Z 2025-10-10T02:00:51.3969158Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:51.3969594Z 2025-10-10T02:00:51.3970297Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:51.3970907Z 2025-10-10T02:00:51.3971402Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:51.3971752Z 2025-10-10T02:00:51.3972053Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3972343Z 2025-10-10T02:00:51.3972836Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:51.3973301Z 2025-10-10T02:00:51.3973568Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:00:51.3973846Z 2025-10-10T02:00:51.3974099Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3974356Z 2025-10-10T02:00:51.3974947Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:51.3975462Z 2025-10-10T02:00:51.3975760Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:51.3976048Z 2025-10-10T02:00:51.3976323Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3976593Z 2025-10-10T02:00:51.3977201Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:51.3977690Z 2025-10-10T02:00:51.3977991Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:00:51.3978300Z 2025-10-10T02:00:51.3978576Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3978857Z 2025-10-10T02:00:51.3979344Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:51.3979777Z 2025-10-10T02:00:51.3980134Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:51.3980469Z 2025-10-10T02:00:51.3980764Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3981042Z 2025-10-10T02:00:51.3981390Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:00:51.3981720Z 2025-10-10T02:00:51.3982084Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:00:51.3982422Z 2025-10-10T02:00:51.3982807Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:51.3983174Z 2025-10-10T02:00:51.3983520Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:00:51.3983860Z 2025-10-10T02:00:51.3984227Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:00:51.3984585Z 2025-10-10T02:00:51.3984959Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:51.3985317Z 2025-10-10T02:00:51.3985826Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:51.3986293Z 2025-10-10T02:00:51.3986652Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:51.3986993Z 2025-10-10T02:00:51.3987520Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:51.3987997Z 2025-10-10T02:00:51.3988375Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:51.3988730Z 2025-10-10T02:00:51.3989331Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:51.3989865Z 2025-10-10T02:00:51.3990214Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:00:51.3990523Z 2025-10-10T02:00:51.3991159Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:51.3991748Z 2025-10-10T02:00:51.3992159Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:51.3992516Z 2025-10-10T02:00:51.3992771Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3993030Z 2025-10-10T02:00:51.3993601Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:51.3994122Z 2025-10-10T02:00:51.3994464Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:00:51.3994759Z 2025-10-10T02:00:51.3995038Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3995308Z 2025-10-10T02:00:51.3995924Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:51.3996854Z 2025-10-10T02:00:51.3997167Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:00:51.3997465Z 2025-10-10T02:00:51.3997735Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.3998009Z 2025-10-10T02:00:51.3998601Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:51.3999232Z 2025-10-10T02:00:51.3999561Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:51.3999883Z 2025-10-10T02:00:51.4000161Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4000434Z 2025-10-10T02:00:51.4000966Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:51.4001476Z 2025-10-10T02:00:51.4001779Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:00:51.4002077Z 2025-10-10T02:00:51.4002354Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4002642Z 2025-10-10T02:00:51.4003191Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:51.4003694Z 2025-10-10T02:00:51.4004010Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:00:51.4004325Z 2025-10-10T02:00:51.4004553Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:00:51.4004803Z 2025-10-10T02:00:51.4005385Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:51.4005907Z 2025-10-10T02:00:51.4006192Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] raise RuntimeError( 2025-10-10T02:00:51.4006458Z 2025-10-10T02:00:51.4007030Z (EngineCore_DP0 pid=12010) ERROR 10-10 02:00:51 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:51.4007845Z (EngineCore_DP0 pid=12010) Process EngineCore_DP0: 2025-10-10T02:00:51.4008246Z (EngineCore_DP0 pid=12010) Traceback (most recent call last): 2025-10-10T02:00:51.4008860Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:00:51.4009385Z (EngineCore_DP0 pid=12010) self.run() 2025-10-10T02:00:51.4009922Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:00:51.4010557Z (EngineCore_DP0 pid=12010) self._target(*self._args, **self._kwargs) 2025-10-10T02:00:51.4011240Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:00:51.4011792Z (EngineCore_DP0 pid=12010) raise e 2025-10-10T02:00:51.4012392Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:00:51.4013052Z (EngineCore_DP0 pid=12010) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:00:51.4013503Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4014145Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:00:51.4014800Z (EngineCore_DP0 pid=12010) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:00:51.4015465Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:00:51.4016092Z (EngineCore_DP0 pid=12010) self.model_executor = executor_class(vllm_config) 2025-10-10T02:00:51.4016558Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4017221Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:00:51.4017814Z (EngineCore_DP0 pid=12010) self._init_executor() 2025-10-10T02:00:51.4018496Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:00:51.4019197Z (EngineCore_DP0 pid=12010) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:00:51.4019930Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:00:51.4020649Z (EngineCore_DP0 pid=12010) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:00:51.4021153Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4021792Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:00:51.4022384Z (EngineCore_DP0 pid=12010) return func(*args, **kwargs) 2025-10-10T02:00:51.4022774Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4023460Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:00:51.4024142Z (EngineCore_DP0 pid=12010) worker_class = resolve_obj_by_qualname( 2025-10-10T02:00:51.4024570Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4025245Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:00:51.4025967Z (EngineCore_DP0 pid=12010) module = importlib.import_module(module_name) 2025-10-10T02:00:51.4026422Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4027010Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:00:51.4027630Z (EngineCore_DP0 pid=12010) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:00:51.4028167Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4028671Z (EngineCore_DP0 pid=12010) File "", line 1387, in _gcd_import 2025-10-10T02:00:51.4029228Z (EngineCore_DP0 pid=12010) File "", line 1360, in _find_and_load 2025-10-10T02:00:51.4029843Z (EngineCore_DP0 pid=12010) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:00:51.4030442Z (EngineCore_DP0 pid=12010) File "", line 935, in _load_unlocked 2025-10-10T02:00:51.4031020Z (EngineCore_DP0 pid=12010) File "", line 999, in exec_module 2025-10-10T02:00:51.4031626Z (EngineCore_DP0 pid=12010) File "", line 488, in _call_with_frames_removed 2025-10-10T02:00:51.4032368Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:00:51.4033052Z (EngineCore_DP0 pid=12010) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:00:51.4033780Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:00:51.4034510Z (EngineCore_DP0 pid=12010) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:00:51.4035309Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:00:51.4035987Z (EngineCore_DP0 pid=12010) class FlashAttentionMetadataBuilder( 2025-10-10T02:00:51.4036817Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:00:51.4037653Z (EngineCore_DP0 pid=12010) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:00:51.4038159Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4038854Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:00:51.4039637Z (EngineCore_DP0 pid=12010) if not is_fa_version_supported(fa_version): 2025-10-10T02:00:51.4040081Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4040881Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:00:51.4041630Z (EngineCore_DP0 pid=12010) return _is_fa2_supported(device)[0] 2025-10-10T02:00:51.4042047Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4042769Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:00:51.4043557Z (EngineCore_DP0 pid=12010) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:00:51.4044021Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4044700Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:00:51.4045339Z (EngineCore_DP0 pid=12010) prop = get_device_properties(device) 2025-10-10T02:00:51.4045766Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:00:51.4046495Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:00:51.4047193Z (EngineCore_DP0 pid=12010) _lazy_init() # will define _get_device_properties 2025-10-10T02:00:51.4047599Z (EngineCore_DP0 pid=12010) ^^^^^^^^^^^^ 2025-10-10T02:00:51.4048191Z (EngineCore_DP0 pid=12010) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:00:51.4048765Z (EngineCore_DP0 pid=12010) raise RuntimeError( 2025-10-10T02:00:51.4049454Z (EngineCore_DP0 pid=12010) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:00:51.8113223Z FAILED 2025-10-10T02:00:51.8244021Z models/test_initialization.py::test_can_initialize_large_subset[MolmoForCausalLM] Fork a new process to run a test 12014 2025-10-10T02:00:51.8255804Z Fork a new process to run a test 0 2025-10-10T02:00:51.8259700Z `transformers==4.56.2` installed, but `transformers<=4.48` is required to run this model. Reason: Incorrectly-detected `tensorflow` import. 2025-10-10T02:00:52.1340264Z PASSED 2025-10-10T02:00:52.1470638Z models/test_initialization.py::test_can_initialize_large_subset[Grok1ModelForCausalLM] Fork a new process to run a test 12015 2025-10-10T02:00:52.1482240Z Fork a new process to run a test 0 2025-10-10T02:00:52.1756742Z INFO 10-10 02:00:52 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Grok1ModelForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'hpcai-tech/grok-1'} 2025-10-10T02:00:52.4093318Z 2025-10-10T02:00:52.4095497Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:52.4095840Z config.json: 1.00kB [00:00, 4.75MB/s] 2025-10-10T02:00:52.4853758Z 2025-10-10T02:00:52.4854981Z configuration_grok1.py: 0.00B [00:00, ?B/s] 2025-10-10T02:00:52.4855370Z configuration_grok1.py: 2.13kB [00:00, 21.8MB/s] 2025-10-10T02:00:52.4940719Z A new version of the following files was downloaded from https://huggingface.co/hpcai-tech/grok-1: 2025-10-10T02:00:52.4941253Z - configuration_grok1.py 2025-10-10T02:00:52.4941882Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:00:59.6101385Z INFO 10-10 02:00:59 [model.py:551] Resolved architecture: Grok1ModelForCausalLM 2025-10-10T02:00:59.6101898Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:00:59.6349774Z INFO 10-10 02:00:59 [model.py:1545] Using max model len 8192 2025-10-10T02:00:59.6351982Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T02:00:59.7026147Z INFO 10-10 02:00:59 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:00:59.7591360Z 2025-10-10T02:00:59.7592943Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:59.7593331Z tokenizer_config.json: 1.02kB [00:00, 7.60MB/s] 2025-10-10T02:00:59.8931444Z 2025-10-10T02:00:59.9261455Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T02:00:59.9261786Z tokenizer.json: 9.14MB [00:00, 277MB/s] 2025-10-10T02:00:59.9910497Z 2025-10-10T02:00:59.9912253Z special_tokens_map.json: 0% 0.00/555 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. 2025-10-10T02:01:00.5860805Z (EngineCore_DP0 pid=12073) INFO 10-10 02:01:00 [core.py:648] Waiting for init message from front-end. 2025-10-10T02:01:00.5940956Z (EngineCore_DP0 pid=12073) INFO 10-10 02:01:00 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='hpcai-tech/grok-1', speculative_config=None, tokenizer='hpcai-tech/grok-1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=hpcai-tech/grok-1, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:00.7328140Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:00.7328878Z 2025-10-10T02:01:00.7329377Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:00.7329858Z 2025-10-10T02:01:00.7330882Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:00.7331629Z 2025-10-10T02:01:00.7332041Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:00.7332435Z 2025-10-10T02:01:00.7332789Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7333286Z 2025-10-10T02:01:00.7334198Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:00.7334838Z 2025-10-10T02:01:00.7335410Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:00.7335839Z 2025-10-10T02:01:00.7336611Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:00.7337453Z 2025-10-10T02:01:00.7338097Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:00.7338626Z 2025-10-10T02:01:00.7339157Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7339663Z 2025-10-10T02:01:00.7340577Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:00.7341397Z 2025-10-10T02:01:00.7341843Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] self._init_executor() 2025-10-10T02:01:00.7342309Z 2025-10-10T02:01:00.7342909Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:00.7343426Z 2025-10-10T02:01:00.7343752Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:00.7344079Z 2025-10-10T02:01:00.7344619Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:00.7345126Z 2025-10-10T02:01:00.7345471Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:00.7345817Z 2025-10-10T02:01:00.7346122Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7346409Z 2025-10-10T02:01:00.7346908Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:00.7347380Z 2025-10-10T02:01:00.7347653Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:00.7347940Z 2025-10-10T02:01:00.7348201Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7348467Z 2025-10-10T02:01:00.7348987Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:00.7349471Z 2025-10-10T02:01:00.7349830Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:00.7350168Z 2025-10-10T02:01:00.7350447Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7350729Z 2025-10-10T02:01:00.7351262Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:00.7351811Z 2025-10-10T02:01:00.7352128Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:00.7352432Z 2025-10-10T02:01:00.7352715Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7352992Z 2025-10-10T02:01:00.7353445Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:00.7353879Z 2025-10-10T02:01:00.7354275Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:00.7354618Z 2025-10-10T02:01:00.7354911Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7355205Z 2025-10-10T02:01:00.7355567Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:00.7355922Z 2025-10-10T02:01:00.7356279Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:00.7356629Z 2025-10-10T02:01:00.7357012Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:00.7357383Z 2025-10-10T02:01:00.7357743Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:00.7358095Z 2025-10-10T02:01:00.7358489Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:00.7358856Z 2025-10-10T02:01:00.7359386Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:00.7359758Z 2025-10-10T02:01:00.7360278Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:00.7360751Z 2025-10-10T02:01:00.7361115Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:00.7361470Z 2025-10-10T02:01:00.7362000Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:00.7362493Z 2025-10-10T02:01:00.7362870Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:00.7363249Z 2025-10-10T02:01:00.7363811Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:00.7364330Z 2025-10-10T02:01:00.7364628Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:00.7364928Z 2025-10-10T02:01:00.7365615Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:00.7366230Z 2025-10-10T02:01:00.7366618Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:00.7367025Z 2025-10-10T02:01:00.7367291Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7367572Z 2025-10-10T02:01:00.7368155Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:00.7368678Z 2025-10-10T02:01:00.7368982Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:00.7369288Z 2025-10-10T02:01:00.7369659Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7369947Z 2025-10-10T02:01:00.7370553Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:00.7371240Z 2025-10-10T02:01:00.7371545Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:00.7371842Z 2025-10-10T02:01:00.7372106Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7372380Z 2025-10-10T02:01:00.7372966Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:00.7373522Z 2025-10-10T02:01:00.7373847Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:00.7374167Z 2025-10-10T02:01:00.7374452Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7374735Z 2025-10-10T02:01:00.7375268Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:00.7375758Z 2025-10-10T02:01:00.7376054Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:00.7376346Z 2025-10-10T02:01:00.7376620Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7376909Z 2025-10-10T02:01:00.7377442Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:00.7377948Z 2025-10-10T02:01:00.7378267Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:00.7378582Z 2025-10-10T02:01:00.7378814Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:00.7379073Z 2025-10-10T02:01:00.7379565Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:00.7380020Z 2025-10-10T02:01:00.7380328Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] raise RuntimeError( 2025-10-10T02:01:00.7380634Z 2025-10-10T02:01:00.7381218Z (EngineCore_DP0 pid=12073) ERROR 10-10 02:01:00 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:00.7381954Z (EngineCore_DP0 pid=12073) Process EngineCore_DP0: 2025-10-10T02:01:00.7382363Z (EngineCore_DP0 pid=12073) Traceback (most recent call last): 2025-10-10T02:01:00.7383030Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:00.7383554Z (EngineCore_DP0 pid=12073) self.run() 2025-10-10T02:01:00.7384088Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:00.7384661Z (EngineCore_DP0 pid=12073) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:00.7385340Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:00.7385938Z (EngineCore_DP0 pid=12073) raise e 2025-10-10T02:01:00.7386547Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:00.7387198Z (EngineCore_DP0 pid=12073) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:00.7387661Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7388298Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:00.7388942Z (EngineCore_DP0 pid=12073) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:00.7389614Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:00.7390252Z (EngineCore_DP0 pid=12073) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:00.7390721Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7391390Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:00.7391995Z (EngineCore_DP0 pid=12073) self._init_executor() 2025-10-10T02:01:00.7392665Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:00.7393380Z (EngineCore_DP0 pid=12073) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:00.7394113Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:00.7394840Z (EngineCore_DP0 pid=12073) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:00.7395353Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7396009Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:00.7396845Z (EngineCore_DP0 pid=12073) return func(*args, **kwargs) 2025-10-10T02:01:00.7397261Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7397915Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:00.7398644Z (EngineCore_DP0 pid=12073) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:00.7399216Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7399915Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:00.7400592Z (EngineCore_DP0 pid=12073) module = importlib.import_module(module_name) 2025-10-10T02:01:00.7401150Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7401752Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:00.7402380Z (EngineCore_DP0 pid=12073) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:00.7402892Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7403404Z (EngineCore_DP0 pid=12073) File "", line 1387, in _gcd_import 2025-10-10T02:01:00.7404030Z (EngineCore_DP0 pid=12073) File "", line 1360, in _find_and_load 2025-10-10T02:01:00.7404641Z (EngineCore_DP0 pid=12073) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:00.7405232Z (EngineCore_DP0 pid=12073) File "", line 935, in _load_unlocked 2025-10-10T02:01:00.7405818Z (EngineCore_DP0 pid=12073) File "", line 999, in exec_module 2025-10-10T02:01:00.7406441Z (EngineCore_DP0 pid=12073) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:00.7407203Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:00.7407902Z (EngineCore_DP0 pid=12073) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:00.7408634Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:00.7409365Z (EngineCore_DP0 pid=12073) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:00.7410167Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:00.7410848Z (EngineCore_DP0 pid=12073) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:00.7411644Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:00.7412464Z (EngineCore_DP0 pid=12073) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:00.7412949Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7413652Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:00.7414359Z (EngineCore_DP0 pid=12073) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:00.7414811Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7415565Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:00.7416272Z (EngineCore_DP0 pid=12073) return _is_fa2_supported(device)[0] 2025-10-10T02:01:00.7416732Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7417504Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:00.7418241Z (EngineCore_DP0 pid=12073) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:00.7418707Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7419432Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:00.7420072Z (EngineCore_DP0 pid=12073) prop = get_device_properties(device) 2025-10-10T02:01:00.7420501Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:00.7421180Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:00.7421893Z (EngineCore_DP0 pid=12073) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:00.7422314Z (EngineCore_DP0 pid=12073) ^^^^^^^^^^^^ 2025-10-10T02:01:00.7422902Z (EngineCore_DP0 pid=12073) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:00.7423472Z (EngineCore_DP0 pid=12073) raise RuntimeError( 2025-10-10T02:01:00.7424169Z (EngineCore_DP0 pid=12073) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:01.1484065Z FAILED 2025-10-10T02:01:01.1613610Z models/test_initialization.py::test_can_initialize_large_subset[SmolLM3ForCausalLM] Fork a new process to run a test 12077 2025-10-10T02:01:01.1624835Z Fork a new process to run a test 0 2025-10-10T02:01:01.1902767Z INFO 10-10 02:01:01 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='SmolLM3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'HuggingFaceTB/SmolLM3-3B'} 2025-10-10T02:01:01.3381770Z 2025-10-10T02:01:01.3383437Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:01.3383931Z config.json: 1.92kB [00:00, 11.2MB/s] 2025-10-10T02:01:01.4323861Z INFO 10-10 02:01:01 [model.py:551] Resolved architecture: SmolLM3ForCausalLM 2025-10-10T02:01:01.4324602Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:01:01.4573829Z INFO 10-10 02:01:01 [model.py:1545] Using max model len 65536 2025-10-10T02:01:01.6317859Z INFO 10-10 02:01:01 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:01:01.6784242Z 2025-10-10T02:01:01.6787046Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:01.6787656Z tokenizer_config.json: 50.4kB [00:00, 197MB/s] 2025-10-10T02:01:01.8124058Z 2025-10-10T02:01:02.1798093Z tokenizer.json: 0% 0.00/17.2M [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:03.1434660Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:03.1435274Z 2025-10-10T02:01:03.1435811Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:03.1436354Z 2025-10-10T02:01:03.1437325Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:03.1438202Z 2025-10-10T02:01:03.1438774Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:03.1439437Z 2025-10-10T02:01:03.1439991Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1440525Z 2025-10-10T02:01:03.1441453Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:03.1442329Z 2025-10-10T02:01:03.1443033Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:03.1443685Z 2025-10-10T02:01:03.1444671Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:03.1445558Z 2025-10-10T02:01:03.1446562Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:03.1447356Z 2025-10-10T02:01:03.1447976Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1448521Z 2025-10-10T02:01:03.1449546Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:03.1450657Z 2025-10-10T02:01:03.1451173Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] self._init_executor() 2025-10-10T02:01:03.1451671Z 2025-10-10T02:01:03.1452660Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:03.1453470Z 2025-10-10T02:01:03.1454054Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:03.1454647Z 2025-10-10T02:01:03.1455806Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:03.1456438Z 2025-10-10T02:01:03.1456836Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:03.1457195Z 2025-10-10T02:01:03.1457498Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1457810Z 2025-10-10T02:01:03.1458312Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:03.1458797Z 2025-10-10T02:01:03.1459079Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:03.1459360Z 2025-10-10T02:01:03.1459626Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1459893Z 2025-10-10T02:01:03.1460440Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:03.1460941Z 2025-10-10T02:01:03.1461242Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:03.1461536Z 2025-10-10T02:01:03.1461813Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1462095Z 2025-10-10T02:01:03.1462631Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:03.1463129Z 2025-10-10T02:01:03.1463443Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:03.1463749Z 2025-10-10T02:01:03.1464025Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1464306Z 2025-10-10T02:01:03.1464766Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:03.1465186Z 2025-10-10T02:01:03.1465538Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:03.1465874Z 2025-10-10T02:01:03.1466233Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1466567Z 2025-10-10T02:01:03.1466925Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:03.1467263Z 2025-10-10T02:01:03.1467619Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:03.1468028Z 2025-10-10T02:01:03.1468409Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:03.1468777Z 2025-10-10T02:01:03.1469122Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:03.1469467Z 2025-10-10T02:01:03.1469833Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:03.1470210Z 2025-10-10T02:01:03.1470647Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:03.1471015Z 2025-10-10T02:01:03.1471524Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:03.1471995Z 2025-10-10T02:01:03.1472357Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:03.1472699Z 2025-10-10T02:01:03.1473228Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:03.1473714Z 2025-10-10T02:01:03.1474092Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:03.1474466Z 2025-10-10T02:01:03.1475038Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:03.1475556Z 2025-10-10T02:01:03.1475854Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:03.1476173Z 2025-10-10T02:01:03.1476830Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:03.1477424Z 2025-10-10T02:01:03.1477798Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:03.1478157Z 2025-10-10T02:01:03.1478424Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1478689Z 2025-10-10T02:01:03.1479401Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:03.1479929Z 2025-10-10T02:01:03.1480246Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:03.1480547Z 2025-10-10T02:01:03.1480829Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1481106Z 2025-10-10T02:01:03.1481764Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:03.1482367Z 2025-10-10T02:01:03.1482662Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:03.1482956Z 2025-10-10T02:01:03.1483222Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1483561Z 2025-10-10T02:01:03.1484144Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:03.1484689Z 2025-10-10T02:01:03.1485016Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:03.1485332Z 2025-10-10T02:01:03.1485615Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1485896Z 2025-10-10T02:01:03.1486473Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:03.1486971Z 2025-10-10T02:01:03.1487265Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:03.1487560Z 2025-10-10T02:01:03.1487832Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1488118Z 2025-10-10T02:01:03.1488642Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:03.1489147Z 2025-10-10T02:01:03.1489466Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:03.1489786Z 2025-10-10T02:01:03.1490021Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:03.1490268Z 2025-10-10T02:01:03.1490776Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:03.1491240Z 2025-10-10T02:01:03.1491500Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] raise RuntimeError( 2025-10-10T02:01:03.1491765Z 2025-10-10T02:01:03.1492358Z (EngineCore_DP0 pid=12105) ERROR 10-10 02:01:03 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:03.1493090Z (EngineCore_DP0 pid=12105) Process EngineCore_DP0: 2025-10-10T02:01:03.1493502Z (EngineCore_DP0 pid=12105) Traceback (most recent call last): 2025-10-10T02:01:03.1494117Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:03.1494658Z (EngineCore_DP0 pid=12105) self.run() 2025-10-10T02:01:03.1495199Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:03.1495780Z (EngineCore_DP0 pid=12105) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:03.1496812Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:03.1497394Z (EngineCore_DP0 pid=12105) raise e 2025-10-10T02:01:03.1498124Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:03.1498886Z (EngineCore_DP0 pid=12105) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:03.1499358Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1499998Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:03.1500718Z (EngineCore_DP0 pid=12105) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:03.1501388Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:03.1502023Z (EngineCore_DP0 pid=12105) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:03.1502487Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1503156Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:03.1503838Z (EngineCore_DP0 pid=12105) self._init_executor() 2025-10-10T02:01:03.1504521Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:03.1505244Z (EngineCore_DP0 pid=12105) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:03.1505983Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:03.1506704Z (EngineCore_DP0 pid=12105) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:03.1507205Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1507857Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:03.1508455Z (EngineCore_DP0 pid=12105) return func(*args, **kwargs) 2025-10-10T02:01:03.1508851Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1509505Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:03.1510166Z (EngineCore_DP0 pid=12105) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:03.1510605Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1511288Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:03.1511959Z (EngineCore_DP0 pid=12105) module = importlib.import_module(module_name) 2025-10-10T02:01:03.1512419Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1513015Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:03.1513644Z (EngineCore_DP0 pid=12105) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:03.1514145Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1514653Z (EngineCore_DP0 pid=12105) File "", line 1387, in _gcd_import 2025-10-10T02:01:03.1515223Z (EngineCore_DP0 pid=12105) File "", line 1360, in _find_and_load 2025-10-10T02:01:03.1515880Z (EngineCore_DP0 pid=12105) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:03.1516510Z (EngineCore_DP0 pid=12105) File "", line 935, in _load_unlocked 2025-10-10T02:01:03.1517100Z (EngineCore_DP0 pid=12105) File "", line 999, in exec_module 2025-10-10T02:01:03.1517728Z (EngineCore_DP0 pid=12105) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:03.1518540Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:03.1519299Z (EngineCore_DP0 pid=12105) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:03.1520044Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:03.1520780Z (EngineCore_DP0 pid=12105) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:03.1521626Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:03.1522313Z (EngineCore_DP0 pid=12105) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:03.1523123Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:03.1523967Z (EngineCore_DP0 pid=12105) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:03.1524453Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1525163Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:03.1525862Z (EngineCore_DP0 pid=12105) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:03.1526317Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1527087Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:03.1527802Z (EngineCore_DP0 pid=12105) return _is_fa2_supported(device)[0] 2025-10-10T02:01:03.1528221Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1528948Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:03.1529675Z (EngineCore_DP0 pid=12105) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:03.1530141Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1530823Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:03.1531461Z (EngineCore_DP0 pid=12105) prop = get_device_properties(device) 2025-10-10T02:01:03.1531885Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:03.1532554Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:03.1533240Z (EngineCore_DP0 pid=12105) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:03.1533649Z (EngineCore_DP0 pid=12105) ^^^^^^^^^^^^ 2025-10-10T02:01:03.1534278Z (EngineCore_DP0 pid=12105) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:03.1534900Z (EngineCore_DP0 pid=12105) raise RuntimeError( 2025-10-10T02:01:03.1535597Z (EngineCore_DP0 pid=12105) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:03.5617503Z FAILED 2025-10-10T02:01:03.5746013Z models/test_initialization.py::test_can_initialize_large_subset[Mamba2ForCausalLM] Fork a new process to run a test 12109 2025-10-10T02:01:03.5757738Z Fork a new process to run a test 0 2025-10-10T02:01:03.6039419Z INFO 10-10 02:01:03 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Mamba2ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'mistralai/Mamba-Codestral-7B-v0.1'} 2025-10-10T02:01:03.7185448Z 2025-10-10T02:01:03.7187324Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:03.7187816Z config.json: 1.01kB [00:00, 5.94MB/s] 2025-10-10T02:01:10.4216537Z INFO 10-10 02:01:10 [model.py:551] Resolved architecture: Mamba2ForCausalLM 2025-10-10T02:01:10.4217031Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:01:10.4466397Z WARNING 10-10 02:01:10 [model.py:1867] The model's config.json does not contain any of the following keys to determine the original maximum length of the model: ['max_position_embeddings', 'n_positions', 'max_seq_len', 'seq_length', 'model_max_length', 'max_target_positions', 'max_sequence_length', 'max_seq_length', 'seq_len']. Assuming the model's maximum length is 2048. 2025-10-10T02:01:10.4467780Z INFO 10-10 02:01:10 [model.py:1545] Using max model len 2048 2025-10-10T02:01:10.6554318Z INFO 10-10 02:01:10 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:01:10.6555615Z INFO 10-10 02:01:10 [config.py:297] Hybrid or mamba-based model detected: disabling prefix caching since it is not yet supported. 2025-10-10T02:01:10.6556557Z INFO 10-10 02:01:10 [config.py:308] Hybrid or mamba-based model detected: setting cudagraph mode to FULL_AND_PIECEWISE in order to optimize performance. 2025-10-10T02:01:10.7588923Z 2025-10-10T02:01:10.7595010Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:10.7595363Z tokenizer_config.json: 137kB [00:00, 205MB/s] 2025-10-10T02:01:10.9029934Z 2025-10-10T02:01:11.2099278Z tokenizer.model: 0% 0.00/588k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:12.0537518Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:12.0538000Z 2025-10-10T02:01:12.0538421Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:12.0538730Z 2025-10-10T02:01:12.0539316Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:12.0539826Z 2025-10-10T02:01:12.0540151Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:12.0540457Z 2025-10-10T02:01:12.0540747Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0541035Z 2025-10-10T02:01:12.0541520Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:12.0542001Z 2025-10-10T02:01:12.0542333Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:12.0542666Z 2025-10-10T02:01:12.0543149Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:12.0543601Z 2025-10-10T02:01:12.0544065Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:12.0544382Z 2025-10-10T02:01:12.0544679Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0544961Z 2025-10-10T02:01:12.0545474Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:12.0545942Z 2025-10-10T02:01:12.0546200Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] self._init_executor() 2025-10-10T02:01:12.0546699Z 2025-10-10T02:01:12.0547299Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:12.0548136Z 2025-10-10T02:01:12.0548527Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:12.0548858Z 2025-10-10T02:01:12.0549525Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:12.0550029Z 2025-10-10T02:01:12.0550376Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:12.0550717Z 2025-10-10T02:01:12.0551016Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0551311Z 2025-10-10T02:01:12.0552168Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:12.0552659Z 2025-10-10T02:01:12.0552963Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:12.0553280Z 2025-10-10T02:01:12.0553545Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0553814Z 2025-10-10T02:01:12.0554339Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:12.0554824Z 2025-10-10T02:01:12.0555118Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:12.0555417Z 2025-10-10T02:01:12.0555699Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0555976Z 2025-10-10T02:01:12.0556514Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:12.0557004Z 2025-10-10T02:01:12.0557318Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:12.0557622Z 2025-10-10T02:01:12.0557899Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0558194Z 2025-10-10T02:01:12.0558644Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:12.0559184Z 2025-10-10T02:01:12.0559535Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:12.0559871Z 2025-10-10T02:01:12.0560160Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0560451Z 2025-10-10T02:01:12.0560806Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:12.0561140Z 2025-10-10T02:01:12.0561499Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:12.0561846Z 2025-10-10T02:01:12.0562314Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:12.0562724Z 2025-10-10T02:01:12.0563085Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:12.0563431Z 2025-10-10T02:01:12.0563803Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:12.0564165Z 2025-10-10T02:01:12.0564608Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:12.0564983Z 2025-10-10T02:01:12.0565496Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:12.0565980Z 2025-10-10T02:01:12.0566337Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:12.0566682Z 2025-10-10T02:01:12.0567250Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:12.0567773Z 2025-10-10T02:01:12.0568184Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:12.0568560Z 2025-10-10T02:01:12.0569134Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:12.0569644Z 2025-10-10T02:01:12.0569953Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:12.0570251Z 2025-10-10T02:01:12.0570896Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:12.0571475Z 2025-10-10T02:01:12.0571847Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:12.0572208Z 2025-10-10T02:01:12.0572469Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0572753Z 2025-10-10T02:01:12.0573317Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:12.0573841Z 2025-10-10T02:01:12.0574145Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:12.0574453Z 2025-10-10T02:01:12.0574732Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0575008Z 2025-10-10T02:01:12.0575616Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:12.0576168Z 2025-10-10T02:01:12.0576459Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:12.0576743Z 2025-10-10T02:01:12.0577013Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0577286Z 2025-10-10T02:01:12.0577913Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:12.0578494Z 2025-10-10T02:01:12.0578826Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:12.0579147Z 2025-10-10T02:01:12.0579427Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0579751Z 2025-10-10T02:01:12.0580282Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:12.0580772Z 2025-10-10T02:01:12.0581068Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:12.0581360Z 2025-10-10T02:01:12.0581637Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0581913Z 2025-10-10T02:01:12.0582480Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:12.0582968Z 2025-10-10T02:01:12.0583288Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:12.0583597Z 2025-10-10T02:01:12.0583825Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:12.0584077Z 2025-10-10T02:01:12.0584563Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:12.0585020Z 2025-10-10T02:01:12.0585272Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] raise RuntimeError( 2025-10-10T02:01:12.0585542Z 2025-10-10T02:01:12.0586111Z (EngineCore_DP0 pid=12187) ERROR 10-10 02:01:12 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:12.0586817Z (EngineCore_DP0 pid=12187) Process EngineCore_DP0: 2025-10-10T02:01:12.0587221Z (EngineCore_DP0 pid=12187) Traceback (most recent call last): 2025-10-10T02:01:12.0587841Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:12.0588359Z (EngineCore_DP0 pid=12187) self.run() 2025-10-10T02:01:12.0588903Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:12.0589477Z (EngineCore_DP0 pid=12187) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:12.0590159Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:12.0590718Z (EngineCore_DP0 pid=12187) raise e 2025-10-10T02:01:12.0591336Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:12.0591975Z (EngineCore_DP0 pid=12187) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:12.0592448Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0593083Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:12.0593735Z (EngineCore_DP0 pid=12187) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:12.0594451Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:12.0595122Z (EngineCore_DP0 pid=12187) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:12.0606707Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0607456Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:12.0608331Z (EngineCore_DP0 pid=12187) self._init_executor() 2025-10-10T02:01:12.0609159Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:12.0609995Z (EngineCore_DP0 pid=12187) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:12.0610786Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:12.0611587Z (EngineCore_DP0 pid=12187) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:12.0612109Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0612775Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:12.0613383Z (EngineCore_DP0 pid=12187) return func(*args, **kwargs) 2025-10-10T02:01:12.0613784Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0614434Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:12.0615082Z (EngineCore_DP0 pid=12187) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:12.0615514Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0616200Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:12.0616868Z (EngineCore_DP0 pid=12187) module = importlib.import_module(module_name) 2025-10-10T02:01:12.0617325Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0617928Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:12.0618574Z (EngineCore_DP0 pid=12187) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:12.0619072Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0619584Z (EngineCore_DP0 pid=12187) File "", line 1387, in _gcd_import 2025-10-10T02:01:12.0620156Z (EngineCore_DP0 pid=12187) File "", line 1360, in _find_and_load 2025-10-10T02:01:12.0620752Z (EngineCore_DP0 pid=12187) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:12.0621344Z (EngineCore_DP0 pid=12187) File "", line 935, in _load_unlocked 2025-10-10T02:01:12.0621926Z (EngineCore_DP0 pid=12187) File "", line 999, in exec_module 2025-10-10T02:01:12.0622546Z (EngineCore_DP0 pid=12187) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:12.0623373Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:12.0624145Z (EngineCore_DP0 pid=12187) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:12.0624907Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:12.0625645Z (EngineCore_DP0 pid=12187) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:12.0626487Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:12.0627168Z (EngineCore_DP0 pid=12187) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:12.0627970Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:12.0628792Z (EngineCore_DP0 pid=12187) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:12.0629324Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0630038Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:12.0630731Z (EngineCore_DP0 pid=12187) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:12.0631194Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0631949Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:12.0632680Z (EngineCore_DP0 pid=12187) return _is_fa2_supported(device)[0] 2025-10-10T02:01:12.0633103Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0633834Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:12.0634564Z (EngineCore_DP0 pid=12187) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:12.0635029Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0635711Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:12.0636350Z (EngineCore_DP0 pid=12187) prop = get_device_properties(device) 2025-10-10T02:01:12.0636791Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:12.0637467Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:12.0638263Z (EngineCore_DP0 pid=12187) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:12.0638748Z (EngineCore_DP0 pid=12187) ^^^^^^^^^^^^ 2025-10-10T02:01:12.0639551Z (EngineCore_DP0 pid=12187) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:12.0640134Z (EngineCore_DP0 pid=12187) raise RuntimeError( 2025-10-10T02:01:12.0640841Z (EngineCore_DP0 pid=12187) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:12.4674505Z FAILED 2025-10-10T02:01:12.4803318Z models/test_initialization.py::test_can_initialize_large_subset[FalconForCausalLM] Fork a new process to run a test 12191 2025-10-10T02:01:12.4815495Z Fork a new process to run a test 0 2025-10-10T02:01:12.5091724Z INFO 10-10 02:01:12 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='FalconForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'tiiuae/falcon-7b'} 2025-10-10T02:01:12.6178606Z 2025-10-10T02:01:12.6180818Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:12.6181357Z config.json: 1.05kB [00:00, 5.48MB/s] 2025-10-10T02:01:12.7342145Z INFO 10-10 02:01:12 [model.py:551] Resolved architecture: FalconForCausalLM 2025-10-10T02:01:12.7342613Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:01:12.7593848Z INFO 10-10 02:01:12 [model.py:1545] Using max model len 2048 2025-10-10T02:01:12.9475008Z INFO 10-10 02:01:12 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:01:12.9906194Z 2025-10-10T02:01:12.9907453Z tokenizer_config.json: 0% 0.00/287 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:13.6264387Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:13.6265124Z 2025-10-10T02:01:13.6265582Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:13.6265949Z 2025-10-10T02:01:13.6266642Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:13.6267357Z 2025-10-10T02:01:13.6267753Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:13.6268136Z 2025-10-10T02:01:13.6268483Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6268840Z 2025-10-10T02:01:13.6269447Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:13.6269993Z 2025-10-10T02:01:13.6270423Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:13.6270766Z 2025-10-10T02:01:13.6271253Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:13.6271703Z 2025-10-10T02:01:13.6272018Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:13.6272324Z 2025-10-10T02:01:13.6272609Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6272901Z 2025-10-10T02:01:13.6273452Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:13.6273943Z 2025-10-10T02:01:13.6274215Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] self._init_executor() 2025-10-10T02:01:13.6274659Z 2025-10-10T02:01:13.6275359Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:13.6276081Z 2025-10-10T02:01:13.6276622Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:13.6277071Z 2025-10-10T02:01:13.6277777Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:13.6278297Z 2025-10-10T02:01:13.6278676Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:13.6279035Z 2025-10-10T02:01:13.6279447Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6279739Z 2025-10-10T02:01:13.6280243Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:13.6280705Z 2025-10-10T02:01:13.6280982Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:13.6281252Z 2025-10-10T02:01:13.6281513Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6281771Z 2025-10-10T02:01:13.6282385Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:13.6282917Z 2025-10-10T02:01:13.6283227Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:13.6283529Z 2025-10-10T02:01:13.6283798Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6284118Z 2025-10-10T02:01:13.6284644Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:13.6285138Z 2025-10-10T02:01:13.6285437Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:13.6285742Z 2025-10-10T02:01:13.6286014Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6286288Z 2025-10-10T02:01:13.6286787Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:13.6287207Z 2025-10-10T02:01:13.6287555Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:13.6287888Z 2025-10-10T02:01:13.6288181Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6288458Z 2025-10-10T02:01:13.6288803Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:13.6289131Z 2025-10-10T02:01:13.6289489Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:13.6289831Z 2025-10-10T02:01:13.6290209Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:13.6290575Z 2025-10-10T02:01:13.6290915Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:13.6291252Z 2025-10-10T02:01:13.6291611Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:13.6291959Z 2025-10-10T02:01:13.6292337Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:13.6292706Z 2025-10-10T02:01:13.6293214Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:13.6293686Z 2025-10-10T02:01:13.6294042Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:13.6294381Z 2025-10-10T02:01:13.6294903Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:13.6295384Z 2025-10-10T02:01:13.6295760Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:13.6296355Z 2025-10-10T02:01:13.6297034Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:13.6297630Z 2025-10-10T02:01:13.6297950Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:13.6298262Z 2025-10-10T02:01:13.6298909Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:13.6299563Z 2025-10-10T02:01:13.6299950Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:13.6300303Z 2025-10-10T02:01:13.6300562Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6300824Z 2025-10-10T02:01:13.6301394Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:13.6302003Z 2025-10-10T02:01:13.6302315Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:13.6302623Z 2025-10-10T02:01:13.6302916Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6303201Z 2025-10-10T02:01:13.6303819Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:13.6304383Z 2025-10-10T02:01:13.6304669Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:13.6304967Z 2025-10-10T02:01:13.6305230Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6305505Z 2025-10-10T02:01:13.6306084Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:13.6306624Z 2025-10-10T02:01:13.6306944Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:13.6307262Z 2025-10-10T02:01:13.6307542Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6307817Z 2025-10-10T02:01:13.6308354Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:13.6308839Z 2025-10-10T02:01:13.6309133Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:13.6309431Z 2025-10-10T02:01:13.6309709Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6309988Z 2025-10-10T02:01:13.6310512Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:13.6311006Z 2025-10-10T02:01:13.6311320Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:13.6311631Z 2025-10-10T02:01:13.6311857Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:13.6312099Z 2025-10-10T02:01:13.6312641Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:13.6313166Z 2025-10-10T02:01:13.6313425Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] raise RuntimeError( 2025-10-10T02:01:13.6313685Z 2025-10-10T02:01:13.6314261Z (EngineCore_DP0 pid=12199) ERROR 10-10 02:01:13 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:13.6315006Z (EngineCore_DP0 pid=12199) Process EngineCore_DP0: 2025-10-10T02:01:13.6315438Z (EngineCore_DP0 pid=12199) Traceback (most recent call last): 2025-10-10T02:01:13.6316044Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:13.6316569Z (EngineCore_DP0 pid=12199) self.run() 2025-10-10T02:01:13.6317150Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:13.6317748Z (EngineCore_DP0 pid=12199) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:13.6318427Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:13.6318973Z (EngineCore_DP0 pid=12199) raise e 2025-10-10T02:01:13.6319658Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:13.6320294Z (EngineCore_DP0 pid=12199) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:13.6320738Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6321367Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:13.6322004Z (EngineCore_DP0 pid=12199) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:13.6322660Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:13.6323297Z (EngineCore_DP0 pid=12199) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:13.6323752Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6324401Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:13.6324982Z (EngineCore_DP0 pid=12199) self._init_executor() 2025-10-10T02:01:13.6325650Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:13.6326358Z (EngineCore_DP0 pid=12199) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:13.6327081Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:13.6327796Z (EngineCore_DP0 pid=12199) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:13.6328288Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6328940Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:13.6329591Z (EngineCore_DP0 pid=12199) return func(*args, **kwargs) 2025-10-10T02:01:13.6329987Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6330670Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:13.6331306Z (EngineCore_DP0 pid=12199) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:13.6331739Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6332459Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:13.6333120Z (EngineCore_DP0 pid=12199) module = importlib.import_module(module_name) 2025-10-10T02:01:13.6333565Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6334152Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:13.6334826Z (EngineCore_DP0 pid=12199) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:13.6335327Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6335824Z (EngineCore_DP0 pid=12199) File "", line 1387, in _gcd_import 2025-10-10T02:01:13.6336380Z (EngineCore_DP0 pid=12199) File "", line 1360, in _find_and_load 2025-10-10T02:01:13.6336981Z (EngineCore_DP0 pid=12199) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:13.6337558Z (EngineCore_DP0 pid=12199) File "", line 935, in _load_unlocked 2025-10-10T02:01:13.6338132Z (EngineCore_DP0 pid=12199) File "", line 999, in exec_module 2025-10-10T02:01:13.6338736Z (EngineCore_DP0 pid=12199) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:13.6339467Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:13.6340155Z (EngineCore_DP0 pid=12199) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:13.6340882Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:13.6341642Z (EngineCore_DP0 pid=12199) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:13.6342448Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:13.6343133Z (EngineCore_DP0 pid=12199) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:13.6343948Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:13.6344767Z (EngineCore_DP0 pid=12199) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:13.6345253Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6345952Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:13.6346648Z (EngineCore_DP0 pid=12199) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:13.6347120Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6347928Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:13.6348672Z (EngineCore_DP0 pid=12199) return _is_fa2_supported(device)[0] 2025-10-10T02:01:13.6349089Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6349831Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:13.6350619Z (EngineCore_DP0 pid=12199) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:13.6351082Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6351769Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:13.6352411Z (EngineCore_DP0 pid=12199) prop = get_device_properties(device) 2025-10-10T02:01:13.6352873Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:13.6353552Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:13.6354218Z (EngineCore_DP0 pid=12199) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:13.6354614Z (EngineCore_DP0 pid=12199) ^^^^^^^^^^^^ 2025-10-10T02:01:13.6355198Z (EngineCore_DP0 pid=12199) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:13.6355767Z (EngineCore_DP0 pid=12199) raise RuntimeError( 2025-10-10T02:01:13.6356461Z (EngineCore_DP0 pid=12199) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:14.0363350Z FAILED 2025-10-10T02:01:14.0491774Z models/test_initialization.py::test_can_initialize_large_subset[DeepseekV32ForCausalLM] Fork a new process to run a test 12203 2025-10-10T02:01:14.0503627Z Fork a new process to run a test 0 2025-10-10T02:01:14.0782395Z INFO 10-10 02:01:14 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DeepseekV32ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'deepseek-ai/DeepSeek-V3.2-Exp'} 2025-10-10T02:01:14.1887868Z 2025-10-10T02:01:14.1889869Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:14.1890303Z config.json: 1.55kB [00:00, 8.59MB/s] 2025-10-10T02:01:14.2296581Z You are using a model of type deepseek_v32 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. 2025-10-10T02:01:14.2305840Z INFO 10-10 02:01:14 [config.py:617] Detected quantization_config.scale_fmt=ue8m0; enabling Hopper UE8M0. 2025-10-10T02:01:14.2307559Z INFO 10-10 02:01:14 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T02:01:14.3148598Z INFO 10-10 02:01:14 [model.py:551] Resolved architecture: DeepseekV32ForCausalLM 2025-10-10T02:01:14.3149054Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:01:14.3397582Z INFO 10-10 02:01:14 [model.py:1545] Using max model len 163840 2025-10-10T02:01:14.6168405Z INFO 10-10 02:01:14 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:01:14.6169126Z INFO 10-10 02:01:14 [config.py:422] Using custom fp8 kv-cache format for DeepSeekV3.2 2025-10-10T02:01:14.6199946Z INFO 10-10 02:01:14 [cuda.py:176] Forcing kv cache block size to 64 for FlashMLASparse backend. 2025-10-10T02:01:14.6961956Z 2025-10-10T02:01:14.6963372Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:14.6963870Z tokenizer_config.json: 4.06kB [00:00, 16.8MB/s] 2025-10-10T02:01:14.8124371Z 2025-10-10T02:01:14.8400732Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:14.8401087Z tokenizer.json: 7.85MB [00:00, 285MB/s] 2025-10-10T02:01:15.3763060Z 2025-10-10T02:01:15.3764038Z generation_config.json: 0% 0.00/171 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:15.5470751Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:15.5471206Z 2025-10-10T02:01:15.5471807Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:15.5472209Z 2025-10-10T02:01:15.5472900Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:15.5473502Z 2025-10-10T02:01:15.5473893Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:15.5474303Z 2025-10-10T02:01:15.5474665Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5475026Z 2025-10-10T02:01:15.5475647Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:15.5476211Z 2025-10-10T02:01:15.5476887Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:15.5477578Z 2025-10-10T02:01:15.5478215Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:15.5478801Z 2025-10-10T02:01:15.5479344Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:15.5479868Z 2025-10-10T02:01:15.5480235Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5480602Z 2025-10-10T02:01:15.5481363Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:15.5482124Z 2025-10-10T02:01:15.5482541Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] self._init_executor() 2025-10-10T02:01:15.5482959Z 2025-10-10T02:01:15.5484190Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:15.5485070Z 2025-10-10T02:01:15.5485750Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:15.5486117Z 2025-10-10T02:01:15.5486693Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:15.5487222Z 2025-10-10T02:01:15.5487587Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:15.5487939Z 2025-10-10T02:01:15.5488239Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5488525Z 2025-10-10T02:01:15.5489048Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:15.5489512Z 2025-10-10T02:01:15.5489791Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:15.5490068Z 2025-10-10T02:01:15.5490339Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5490602Z 2025-10-10T02:01:15.5491123Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:15.5491638Z 2025-10-10T02:01:15.5491937Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:15.5492248Z 2025-10-10T02:01:15.5492519Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5492803Z 2025-10-10T02:01:15.5493334Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:15.5493833Z 2025-10-10T02:01:15.5494142Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:15.5494447Z 2025-10-10T02:01:15.5494808Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5495089Z 2025-10-10T02:01:15.5495590Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:15.5496020Z 2025-10-10T02:01:15.5496729Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:15.5497077Z 2025-10-10T02:01:15.5497494Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5497803Z 2025-10-10T02:01:15.5498171Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:15.5498526Z 2025-10-10T02:01:15.5498885Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:15.5499238Z 2025-10-10T02:01:15.5499687Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:15.5500066Z 2025-10-10T02:01:15.5500421Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:15.5500769Z 2025-10-10T02:01:15.5501145Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:15.5501515Z 2025-10-10T02:01:15.5501897Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:15.5502280Z 2025-10-10T02:01:15.5502796Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:15.5503276Z 2025-10-10T02:01:15.5503634Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:15.5503981Z 2025-10-10T02:01:15.5504513Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:15.5505008Z 2025-10-10T02:01:15.5505383Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:15.5505749Z 2025-10-10T02:01:15.5506305Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:15.5506816Z 2025-10-10T02:01:15.5507141Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:15.5507455Z 2025-10-10T02:01:15.5508125Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:15.5508710Z 2025-10-10T02:01:15.5509088Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:15.5509471Z 2025-10-10T02:01:15.5509734Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5510009Z 2025-10-10T02:01:15.5510693Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:15.5511283Z 2025-10-10T02:01:15.5511594Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:15.5511898Z 2025-10-10T02:01:15.5512173Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5512448Z 2025-10-10T02:01:15.5513059Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:15.5513670Z 2025-10-10T02:01:15.5513962Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:15.5514250Z 2025-10-10T02:01:15.5514518Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5514791Z 2025-10-10T02:01:15.5515412Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:15.5515950Z 2025-10-10T02:01:15.5516275Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:15.5516612Z 2025-10-10T02:01:15.5516891Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5517172Z 2025-10-10T02:01:15.5517699Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:15.5518188Z 2025-10-10T02:01:15.5518475Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:15.5518768Z 2025-10-10T02:01:15.5519052Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5519430Z 2025-10-10T02:01:15.5519970Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:15.5520463Z 2025-10-10T02:01:15.5520782Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:15.5521101Z 2025-10-10T02:01:15.5521339Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:15.5521591Z 2025-10-10T02:01:15.5522086Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:15.5522549Z 2025-10-10T02:01:15.5522800Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] raise RuntimeError( 2025-10-10T02:01:15.5523070Z 2025-10-10T02:01:15.5523639Z (EngineCore_DP0 pid=12211) ERROR 10-10 02:01:15 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:15.5524341Z (EngineCore_DP0 pid=12211) Process EngineCore_DP0: 2025-10-10T02:01:15.5524748Z (EngineCore_DP0 pid=12211) Traceback (most recent call last): 2025-10-10T02:01:15.5525386Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:15.5525909Z (EngineCore_DP0 pid=12211) self.run() 2025-10-10T02:01:15.5526489Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:15.5527112Z (EngineCore_DP0 pid=12211) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:15.5527808Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:15.5528364Z (EngineCore_DP0 pid=12211) raise e 2025-10-10T02:01:15.5528962Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:15.5529647Z (EngineCore_DP0 pid=12211) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:15.5530103Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5530740Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:15.5531389Z (EngineCore_DP0 pid=12211) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:15.5532103Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:15.5532732Z (EngineCore_DP0 pid=12211) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:15.5533206Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5533882Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:15.5534479Z (EngineCore_DP0 pid=12211) self._init_executor() 2025-10-10T02:01:15.5535154Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:15.5535847Z (EngineCore_DP0 pid=12211) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:15.5536577Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:15.5537314Z (EngineCore_DP0 pid=12211) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:15.5537823Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5538474Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:15.5539068Z (EngineCore_DP0 pid=12211) return func(*args, **kwargs) 2025-10-10T02:01:15.5539467Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5540118Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:15.5540767Z (EngineCore_DP0 pid=12211) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:15.5541204Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5541893Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:15.5542548Z (EngineCore_DP0 pid=12211) module = importlib.import_module(module_name) 2025-10-10T02:01:15.5542999Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5543590Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:15.5544262Z (EngineCore_DP0 pid=12211) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:15.5544804Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5545299Z (EngineCore_DP0 pid=12211) File "", line 1387, in _gcd_import 2025-10-10T02:01:15.5545859Z (EngineCore_DP0 pid=12211) File "", line 1360, in _find_and_load 2025-10-10T02:01:15.5546494Z (EngineCore_DP0 pid=12211) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:15.5547080Z (EngineCore_DP0 pid=12211) File "", line 935, in _load_unlocked 2025-10-10T02:01:15.5547659Z (EngineCore_DP0 pid=12211) File "", line 999, in exec_module 2025-10-10T02:01:15.5548263Z (EngineCore_DP0 pid=12211) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:15.5549061Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:15.5549747Z (EngineCore_DP0 pid=12211) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:15.5550493Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:15.5551234Z (EngineCore_DP0 pid=12211) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:15.5552023Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:15.5552702Z (EngineCore_DP0 pid=12211) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:15.5553501Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:15.5554324Z (EngineCore_DP0 pid=12211) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:15.5554817Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5555518Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:15.5556216Z (EngineCore_DP0 pid=12211) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:15.5556663Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5557414Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:15.5558130Z (EngineCore_DP0 pid=12211) return _is_fa2_supported(device)[0] 2025-10-10T02:01:15.5558557Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5559381Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:15.5560109Z (EngineCore_DP0 pid=12211) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:15.5560564Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5561242Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:15.5561886Z (EngineCore_DP0 pid=12211) prop = get_device_properties(device) 2025-10-10T02:01:15.5562359Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:15.5563063Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:15.5563726Z (EngineCore_DP0 pid=12211) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:15.5564131Z (EngineCore_DP0 pid=12211) ^^^^^^^^^^^^ 2025-10-10T02:01:15.5564787Z (EngineCore_DP0 pid=12211) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:15.5565357Z (EngineCore_DP0 pid=12211) raise RuntimeError( 2025-10-10T02:01:15.5566065Z (EngineCore_DP0 pid=12211) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:15.9623600Z FAILED 2025-10-10T02:01:15.9753597Z models/test_initialization.py::test_can_initialize_large_subset[SolarForCausalLM] Fork a new process to run a test 12215 2025-10-10T02:01:15.9765335Z Fork a new process to run a test 0 2025-10-10T02:01:16.0043621Z INFO 10-10 02:01:16 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='SolarForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'upstage/solar-pro-preview-instruct'} 2025-10-10T02:01:16.1252758Z 2025-10-10T02:01:16.1255106Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:16.1255435Z config.json: 1.03kB [00:00, 4.61MB/s] 2025-10-10T02:01:16.2615334Z 2025-10-10T02:01:16.2617394Z configuration_solar.py: 0.00B [00:00, ?B/s] 2025-10-10T02:01:16.2617776Z configuration_solar.py: 10.1kB [00:00, 57.1MB/s] 2025-10-10T02:01:16.2711242Z A new version of the following files was downloaded from https://huggingface.co/upstage/solar-pro-preview-instruct: 2025-10-10T02:01:16.2711847Z - configuration_solar.py 2025-10-10T02:01:16.2712474Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:01:23.1409528Z INFO 10-10 02:01:23 [model.py:551] Resolved architecture: SolarForCausalLM 2025-10-10T02:01:23.1410029Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:01:23.1657812Z INFO 10-10 02:01:23 [model.py:1545] Using max model len 4096 2025-10-10T02:01:23.1660074Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T02:01:23.2214837Z INFO 10-10 02:01:23 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:01:23.2631222Z 2025-10-10T02:01:23.2633541Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:23.2633913Z tokenizer_config.json: 24.6kB [00:00, 117MB/s] 2025-10-10T02:01:23.3833336Z 2025-10-10T02:01:23.5932699Z tokenizer.model: 0% 0.00/500k [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:24.0405867Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:24.0406372Z 2025-10-10T02:01:24.0406769Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:24.0407139Z 2025-10-10T02:01:24.0407815Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:24.0408416Z 2025-10-10T02:01:24.0408815Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:24.0409211Z 2025-10-10T02:01:24.0409540Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0409822Z 2025-10-10T02:01:24.0410311Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:24.0410758Z 2025-10-10T02:01:24.0411267Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:24.0411599Z 2025-10-10T02:01:24.0412097Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:24.0412554Z 2025-10-10T02:01:24.0413116Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:24.0413520Z 2025-10-10T02:01:24.0413815Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0414221Z 2025-10-10T02:01:24.0415109Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:24.0415750Z 2025-10-10T02:01:24.0416082Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] self._init_executor() 2025-10-10T02:01:24.0416476Z 2025-10-10T02:01:24.0417113Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:24.0417681Z 2025-10-10T02:01:24.0418157Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:24.0418489Z 2025-10-10T02:01:24.0419160Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:24.0419670Z 2025-10-10T02:01:24.0420026Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:24.0420360Z 2025-10-10T02:01:24.0420651Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0420942Z 2025-10-10T02:01:24.0421427Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:24.0421903Z 2025-10-10T02:01:24.0422175Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:24.0422452Z 2025-10-10T02:01:24.0422715Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0422985Z 2025-10-10T02:01:24.0423513Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:24.0423992Z 2025-10-10T02:01:24.0424287Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:24.0424577Z 2025-10-10T02:01:24.0424855Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0425132Z 2025-10-10T02:01:24.0425668Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:24.0426156Z 2025-10-10T02:01:24.0426464Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:24.0426769Z 2025-10-10T02:01:24.0427045Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0427328Z 2025-10-10T02:01:24.0427772Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:24.0428194Z 2025-10-10T02:01:24.0428540Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:24.0428877Z 2025-10-10T02:01:24.0429219Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0429542Z 2025-10-10T02:01:24.0429902Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:24.0430239Z 2025-10-10T02:01:24.0430595Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:24.0430985Z 2025-10-10T02:01:24.0431370Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:24.0431731Z 2025-10-10T02:01:24.0432078Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:24.0432420Z 2025-10-10T02:01:24.0432787Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:24.0433147Z 2025-10-10T02:01:24.0433563Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:24.0433934Z 2025-10-10T02:01:24.0434433Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:24.0434911Z 2025-10-10T02:01:24.0435265Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:24.0435607Z 2025-10-10T02:01:24.0436134Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:24.0436616Z 2025-10-10T02:01:24.0436999Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:24.0437362Z 2025-10-10T02:01:24.0437920Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:24.0438427Z 2025-10-10T02:01:24.0438724Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:24.0439036Z 2025-10-10T02:01:24.0439833Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:24.0440427Z 2025-10-10T02:01:24.0440809Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:24.0441176Z 2025-10-10T02:01:24.0441437Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0441708Z 2025-10-10T02:01:24.0442272Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:24.0442800Z 2025-10-10T02:01:24.0443103Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:24.0443400Z 2025-10-10T02:01:24.0443679Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0443956Z 2025-10-10T02:01:24.0444626Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:24.0445218Z 2025-10-10T02:01:24.0445516Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:24.0445804Z 2025-10-10T02:01:24.0446066Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0446382Z 2025-10-10T02:01:24.0446965Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:24.0447503Z 2025-10-10T02:01:24.0447829Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:24.0448146Z 2025-10-10T02:01:24.0448427Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0448724Z 2025-10-10T02:01:24.0449290Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:24.0449776Z 2025-10-10T02:01:24.0450066Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:24.0450411Z 2025-10-10T02:01:24.0450692Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0450968Z 2025-10-10T02:01:24.0451499Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:24.0451985Z 2025-10-10T02:01:24.0452299Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:24.0452615Z 2025-10-10T02:01:24.0452846Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:24.0453096Z 2025-10-10T02:01:24.0453582Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:24.0454039Z 2025-10-10T02:01:24.0454296Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] raise RuntimeError( 2025-10-10T02:01:24.0454557Z 2025-10-10T02:01:24.0455132Z (EngineCore_DP0 pid=12293) ERROR 10-10 02:01:24 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:24.0455835Z (EngineCore_DP0 pid=12293) Process EngineCore_DP0: 2025-10-10T02:01:24.0456272Z (EngineCore_DP0 pid=12293) Traceback (most recent call last): 2025-10-10T02:01:24.0456887Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:24.0457402Z (EngineCore_DP0 pid=12293) self.run() 2025-10-10T02:01:24.0457940Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:24.0458518Z (EngineCore_DP0 pid=12293) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:24.0459210Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:24.0459771Z (EngineCore_DP0 pid=12293) raise e 2025-10-10T02:01:24.0460417Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:24.0461106Z (EngineCore_DP0 pid=12293) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:24.0461558Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0462199Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:24.0462915Z (EngineCore_DP0 pid=12293) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:24.0463581Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:24.0464214Z (EngineCore_DP0 pid=12293) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:24.0464679Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0465393Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:24.0465995Z (EngineCore_DP0 pid=12293) self._init_executor() 2025-10-10T02:01:24.0466663Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:24.0467363Z (EngineCore_DP0 pid=12293) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:24.0468094Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:24.0468832Z (EngineCore_DP0 pid=12293) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:24.0469350Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0470009Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:24.0470599Z (EngineCore_DP0 pid=12293) return func(*args, **kwargs) 2025-10-10T02:01:24.0470994Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0471665Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:24.0472310Z (EngineCore_DP0 pid=12293) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:24.0472746Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0473424Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:24.0474105Z (EngineCore_DP0 pid=12293) module = importlib.import_module(module_name) 2025-10-10T02:01:24.0474563Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0475154Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:24.0475805Z (EngineCore_DP0 pid=12293) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:24.0476298Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0476804Z (EngineCore_DP0 pid=12293) File "", line 1387, in _gcd_import 2025-10-10T02:01:24.0477375Z (EngineCore_DP0 pid=12293) File "", line 1360, in _find_and_load 2025-10-10T02:01:24.0478030Z (EngineCore_DP0 pid=12293) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:24.0478665Z (EngineCore_DP0 pid=12293) File "", line 935, in _load_unlocked 2025-10-10T02:01:24.0479301Z (EngineCore_DP0 pid=12293) File "", line 999, in exec_module 2025-10-10T02:01:24.0479916Z (EngineCore_DP0 pid=12293) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:24.0480713Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:24.0481413Z (EngineCore_DP0 pid=12293) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:24.0482145Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:24.0482880Z (EngineCore_DP0 pid=12293) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:24.0483702Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:24.0484414Z (EngineCore_DP0 pid=12293) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:24.0485221Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:24.0486053Z (EngineCore_DP0 pid=12293) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:24.0486541Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0487244Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:24.0487949Z (EngineCore_DP0 pid=12293) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:24.0488405Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0489161Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:24.0489871Z (EngineCore_DP0 pid=12293) return _is_fa2_supported(device)[0] 2025-10-10T02:01:24.0490309Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0491035Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:24.0491766Z (EngineCore_DP0 pid=12293) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:24.0492227Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0492907Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:24.0493544Z (EngineCore_DP0 pid=12293) prop = get_device_properties(device) 2025-10-10T02:01:24.0493976Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:24.0494640Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:24.0495320Z (EngineCore_DP0 pid=12293) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:24.0495724Z (EngineCore_DP0 pid=12293) ^^^^^^^^^^^^ 2025-10-10T02:01:24.0496595Z (EngineCore_DP0 pid=12293) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:24.0497264Z (EngineCore_DP0 pid=12293) raise RuntimeError( 2025-10-10T02:01:24.0497979Z (EngineCore_DP0 pid=12293) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:24.4439232Z FAILED 2025-10-10T02:01:24.4568779Z models/test_initialization.py::test_can_initialize_large_subset[MiniCPMV] Fork a new process to run a test 12297 2025-10-10T02:01:24.4580147Z Fork a new process to run a test 0 2025-10-10T02:01:24.4860871Z INFO 10-10 02:01:24 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiniCPMV', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'openbmb/MiniCPM-Llama3-V-2_5'} 2025-10-10T02:01:24.6690926Z 2025-10-10T02:01:24.6693047Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:24.6693652Z config.json: 1.39kB [00:00, 7.11MB/s] 2025-10-10T02:01:24.7414414Z 2025-10-10T02:01:24.7415574Z configuration_minicpm.py: 0.00B [00:00, ?B/s] 2025-10-10T02:01:24.7416111Z configuration_minicpm.py: 4.06kB [00:00, 35.8MB/s] 2025-10-10T02:01:24.7503445Z A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5: 2025-10-10T02:01:24.7504007Z - configuration_minicpm.py 2025-10-10T02:01:24.7504639Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:01:25.0591400Z 2025-10-10T02:01:25.0591794Z preprocessor_config.json: 0% 0.00/599 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:33.1587662Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:33.1588245Z 2025-10-10T02:01:33.1588650Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:33.1589188Z 2025-10-10T02:01:33.1590044Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:33.1590725Z 2025-10-10T02:01:33.1591237Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:33.1591751Z 2025-10-10T02:01:33.1592158Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1592645Z 2025-10-10T02:01:33.1593361Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:33.1594060Z 2025-10-10T02:01:33.1594640Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:33.1595057Z 2025-10-10T02:01:33.1595825Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:33.1596754Z 2025-10-10T02:01:33.1597363Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:33.1598047Z 2025-10-10T02:01:33.1598801Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1599588Z 2025-10-10T02:01:33.1600459Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:33.1600959Z 2025-10-10T02:01:33.1601235Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] self._init_executor() 2025-10-10T02:01:33.1601642Z 2025-10-10T02:01:33.1602212Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:33.1602721Z 2025-10-10T02:01:33.1603044Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:33.1603367Z 2025-10-10T02:01:33.1603911Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:33.1604416Z 2025-10-10T02:01:33.1604840Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:33.1605175Z 2025-10-10T02:01:33.1605468Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1605749Z 2025-10-10T02:01:33.1606240Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:33.1606705Z 2025-10-10T02:01:33.1606982Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:33.1607260Z 2025-10-10T02:01:33.1607528Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1607799Z 2025-10-10T02:01:33.1608347Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:33.1608834Z 2025-10-10T02:01:33.1609125Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:33.1609426Z 2025-10-10T02:01:33.1609693Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1609970Z 2025-10-10T02:01:33.1610495Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:33.1611004Z 2025-10-10T02:01:33.1611311Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:33.1611612Z 2025-10-10T02:01:33.1611896Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1612171Z 2025-10-10T02:01:33.1612615Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:33.1613031Z 2025-10-10T02:01:33.1613385Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:33.1613716Z 2025-10-10T02:01:33.1614003Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1614285Z 2025-10-10T02:01:33.1614672Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:33.1615067Z 2025-10-10T02:01:33.1615424Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:33.1615768Z 2025-10-10T02:01:33.1616142Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:33.1616553Z 2025-10-10T02:01:33.1616897Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:33.1617241Z 2025-10-10T02:01:33.1617605Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:33.1617955Z 2025-10-10T02:01:33.1618337Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:33.1618698Z 2025-10-10T02:01:33.1619247Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:33.1619714Z 2025-10-10T02:01:33.1620070Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:33.1620420Z 2025-10-10T02:01:33.1620944Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:33.1621439Z 2025-10-10T02:01:33.1621819Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:33.1622181Z 2025-10-10T02:01:33.1622738Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:33.1623250Z 2025-10-10T02:01:33.1623543Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:33.1623839Z 2025-10-10T02:01:33.1624480Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:33.1625052Z 2025-10-10T02:01:33.1625427Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:33.1625780Z 2025-10-10T02:01:33.1626043Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1626306Z 2025-10-10T02:01:33.1626874Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:33.1627391Z 2025-10-10T02:01:33.1627689Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:33.1627988Z 2025-10-10T02:01:33.1628259Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1628547Z 2025-10-10T02:01:33.1629144Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:33.1629696Z 2025-10-10T02:01:33.1630020Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:33.1630342Z 2025-10-10T02:01:33.1630609Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1630874Z 2025-10-10T02:01:33.1631452Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:33.1632016Z 2025-10-10T02:01:33.1632340Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:33.1632654Z 2025-10-10T02:01:33.1632933Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1633204Z 2025-10-10T02:01:33.1633727Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:33.1634257Z 2025-10-10T02:01:33.1634548Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:33.1634841Z 2025-10-10T02:01:33.1635108Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1635391Z 2025-10-10T02:01:33.1635919Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:33.1636409Z 2025-10-10T02:01:33.1636721Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:33.1637023Z 2025-10-10T02:01:33.1637267Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:33.1637514Z 2025-10-10T02:01:33.1638002Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:33.1638450Z 2025-10-10T02:01:33.1638700Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] raise RuntimeError( 2025-10-10T02:01:33.1638963Z 2025-10-10T02:01:33.1639615Z (EngineCore_DP0 pid=12355) ERROR 10-10 02:01:33 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:33.1640339Z (EngineCore_DP0 pid=12355) Process EngineCore_DP0: 2025-10-10T02:01:33.1640742Z (EngineCore_DP0 pid=12355) Traceback (most recent call last): 2025-10-10T02:01:33.1641358Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:33.1641882Z (EngineCore_DP0 pid=12355) self.run() 2025-10-10T02:01:33.1642414Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:33.1642979Z (EngineCore_DP0 pid=12355) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:33.1643647Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:33.1644196Z (EngineCore_DP0 pid=12355) raise e 2025-10-10T02:01:33.1644787Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:33.1645474Z (EngineCore_DP0 pid=12355) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:33.1645992Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1646624Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:33.1647267Z (EngineCore_DP0 pid=12355) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:33.1647930Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:33.1648618Z (EngineCore_DP0 pid=12355) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:33.1649080Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1649727Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:33.1650329Z (EngineCore_DP0 pid=12355) self._init_executor() 2025-10-10T02:01:33.1651041Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:33.1651734Z (EngineCore_DP0 pid=12355) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:33.1652460Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:33.1653174Z (EngineCore_DP0 pid=12355) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:33.1653675Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1654322Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:33.1654914Z (EngineCore_DP0 pid=12355) return func(*args, **kwargs) 2025-10-10T02:01:33.1655301Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1655931Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:33.1656561Z (EngineCore_DP0 pid=12355) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:33.1657006Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1657671Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:33.1658328Z (EngineCore_DP0 pid=12355) module = importlib.import_module(module_name) 2025-10-10T02:01:33.1658773Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1659352Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:33.1659976Z (EngineCore_DP0 pid=12355) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:33.1660464Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1660961Z (EngineCore_DP0 pid=12355) File "", line 1387, in _gcd_import 2025-10-10T02:01:33.1661510Z (EngineCore_DP0 pid=12355) File "", line 1360, in _find_and_load 2025-10-10T02:01:33.1662094Z (EngineCore_DP0 pid=12355) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:33.1662719Z (EngineCore_DP0 pid=12355) File "", line 935, in _load_unlocked 2025-10-10T02:01:33.1663345Z (EngineCore_DP0 pid=12355) File "", line 999, in exec_module 2025-10-10T02:01:33.1663953Z (EngineCore_DP0 pid=12355) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:33.1664685Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:33.1665398Z (EngineCore_DP0 pid=12355) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:33.1666120Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:33.1674583Z (EngineCore_DP0 pid=12355) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:33.1675550Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:33.1676349Z (EngineCore_DP0 pid=12355) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:33.1677194Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:33.1678040Z (EngineCore_DP0 pid=12355) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:33.1678536Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1679347Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:33.1680040Z (EngineCore_DP0 pid=12355) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:33.1680508Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1681263Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:33.1681977Z (EngineCore_DP0 pid=12355) return _is_fa2_supported(device)[0] 2025-10-10T02:01:33.1682400Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1683118Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:33.1683843Z (EngineCore_DP0 pid=12355) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:33.1684313Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1684996Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:33.1685642Z (EngineCore_DP0 pid=12355) prop = get_device_properties(device) 2025-10-10T02:01:33.1686076Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:33.1686766Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:33.1687445Z (EngineCore_DP0 pid=12355) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:33.1687846Z (EngineCore_DP0 pid=12355) ^^^^^^^^^^^^ 2025-10-10T02:01:33.1688433Z (EngineCore_DP0 pid=12355) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:33.1689058Z (EngineCore_DP0 pid=12355) raise RuntimeError( 2025-10-10T02:01:33.1689791Z (EngineCore_DP0 pid=12355) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:33.5737071Z FAILED 2025-10-10T02:01:33.5867053Z models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5ForCausalLM] Fork a new process to run a test 12359 2025-10-10T02:01:33.5878331Z Fork a new process to run a test 0 2025-10-10T02:01:33.6153691Z INFO 10-10 02:01:33 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Ernie4_5ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'baidu/ERNIE-4.5-0.3B-PT'} 2025-10-10T02:01:33.7461679Z 2025-10-10T02:01:33.7462602Z config.json: 0% 0.00/633 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:42.8620118Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:42.8620645Z 2025-10-10T02:01:42.8621042Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:42.8621448Z 2025-10-10T02:01:42.8622124Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:42.8622724Z 2025-10-10T02:01:42.8623106Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:42.8623486Z 2025-10-10T02:01:42.8623842Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8624194Z 2025-10-10T02:01:42.8624802Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:42.8625352Z 2025-10-10T02:01:42.8625764Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:42.8626376Z 2025-10-10T02:01:42.8627001Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:42.8627447Z 2025-10-10T02:01:42.8627793Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:42.8628106Z 2025-10-10T02:01:42.8628398Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8628679Z 2025-10-10T02:01:42.8629191Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:42.8629973Z 2025-10-10T02:01:42.8630497Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] self._init_executor() 2025-10-10T02:01:42.8631016Z 2025-10-10T02:01:42.8632095Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:42.8632660Z 2025-10-10T02:01:42.8633083Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:42.8633783Z 2025-10-10T02:01:42.8634426Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:42.8635059Z 2025-10-10T02:01:42.8635439Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:42.8635779Z 2025-10-10T02:01:42.8636177Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8636466Z 2025-10-10T02:01:42.8636975Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:42.8637437Z 2025-10-10T02:01:42.8637719Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:42.8637999Z 2025-10-10T02:01:42.8638251Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8638598Z 2025-10-10T02:01:42.8639309Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:42.8639805Z 2025-10-10T02:01:42.8640101Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:42.8640403Z 2025-10-10T02:01:42.8640680Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8640956Z 2025-10-10T02:01:42.8641498Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:42.8641992Z 2025-10-10T02:01:42.8642306Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:42.8642606Z 2025-10-10T02:01:42.8642891Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8643176Z 2025-10-10T02:01:42.8643623Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:42.8644049Z 2025-10-10T02:01:42.8644391Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:42.8644736Z 2025-10-10T02:01:42.8645027Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8645314Z 2025-10-10T02:01:42.8645658Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:42.8645996Z 2025-10-10T02:01:42.8646355Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:42.8646694Z 2025-10-10T02:01:42.8647070Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:42.8647433Z 2025-10-10T02:01:42.8647787Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:42.8648120Z 2025-10-10T02:01:42.8648487Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:42.8648893Z 2025-10-10T02:01:42.8649310Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:42.8649678Z 2025-10-10T02:01:42.8650173Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:42.8650691Z 2025-10-10T02:01:42.8651045Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:42.8651382Z 2025-10-10T02:01:42.8651895Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:42.8652373Z 2025-10-10T02:01:42.8652744Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:42.8653116Z 2025-10-10T02:01:42.8653701Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:42.8654215Z 2025-10-10T02:01:42.8654514Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:42.8654815Z 2025-10-10T02:01:42.8655437Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:42.8656008Z 2025-10-10T02:01:42.8656389Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:42.8656742Z 2025-10-10T02:01:42.8657000Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8657263Z 2025-10-10T02:01:42.8657831Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:42.8658341Z 2025-10-10T02:01:42.8658647Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:42.8658944Z 2025-10-10T02:01:42.8659215Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8659495Z 2025-10-10T02:01:42.8660093Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:42.8660647Z 2025-10-10T02:01:42.8660929Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:42.8661219Z 2025-10-10T02:01:42.8661479Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8661746Z 2025-10-10T02:01:42.8662325Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:42.8662862Z 2025-10-10T02:01:42.8663184Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:42.8663493Z 2025-10-10T02:01:42.8663818Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8664104Z 2025-10-10T02:01:42.8664691Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:42.8665189Z 2025-10-10T02:01:42.8665475Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:42.8665772Z 2025-10-10T02:01:42.8666091Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8666373Z 2025-10-10T02:01:42.8666914Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:42.8667407Z 2025-10-10T02:01:42.8667726Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:42.8668042Z 2025-10-10T02:01:42.8668284Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:42.8668569Z 2025-10-10T02:01:42.8669061Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:42.8669511Z 2025-10-10T02:01:42.8669764Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] raise RuntimeError( 2025-10-10T02:01:42.8670025Z 2025-10-10T02:01:42.8670587Z (EngineCore_DP0 pid=12438) ERROR 10-10 02:01:42 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:42.8671279Z (EngineCore_DP0 pid=12438) Process EngineCore_DP0: 2025-10-10T02:01:42.8671691Z (EngineCore_DP0 pid=12438) Traceback (most recent call last): 2025-10-10T02:01:42.8672323Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:42.8672841Z (EngineCore_DP0 pid=12438) self.run() 2025-10-10T02:01:42.8673365Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:42.8673941Z (EngineCore_DP0 pid=12438) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:42.8674609Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:42.8675168Z (EngineCore_DP0 pid=12438) raise e 2025-10-10T02:01:42.8675788Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:42.8676430Z (EngineCore_DP0 pid=12438) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:42.8676877Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8677494Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:42.8678136Z (EngineCore_DP0 pid=12438) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:42.8678799Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:42.8679516Z (EngineCore_DP0 pid=12438) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:42.8679970Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8680662Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:42.8681293Z (EngineCore_DP0 pid=12438) self._init_executor() 2025-10-10T02:01:42.8681958Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:42.8682644Z (EngineCore_DP0 pid=12438) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:42.8683435Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:42.8684166Z (EngineCore_DP0 pid=12438) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:42.8684666Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8685308Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:42.8685944Z (EngineCore_DP0 pid=12438) return func(*args, **kwargs) 2025-10-10T02:01:42.8686338Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8686980Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:42.8687612Z (EngineCore_DP0 pid=12438) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:42.8688038Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8688706Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:42.8689412Z (EngineCore_DP0 pid=12438) module = importlib.import_module(module_name) 2025-10-10T02:01:42.8689875Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8690456Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:42.8691084Z (EngineCore_DP0 pid=12438) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:42.8691584Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8692079Z (EngineCore_DP0 pid=12438) File "", line 1387, in _gcd_import 2025-10-10T02:01:42.8692634Z (EngineCore_DP0 pid=12438) File "", line 1360, in _find_and_load 2025-10-10T02:01:42.8693215Z (EngineCore_DP0 pid=12438) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:42.8693798Z (EngineCore_DP0 pid=12438) File "", line 935, in _load_unlocked 2025-10-10T02:01:42.8694376Z (EngineCore_DP0 pid=12438) File "", line 999, in exec_module 2025-10-10T02:01:42.8694984Z (EngineCore_DP0 pid=12438) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:42.8695732Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:42.8696599Z (EngineCore_DP0 pid=12438) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:42.8697325Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:42.8698146Z (EngineCore_DP0 pid=12438) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:42.8698994Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:42.8699661Z (EngineCore_DP0 pid=12438) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:42.8700455Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:42.8701325Z (EngineCore_DP0 pid=12438) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:42.8701811Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8702511Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:42.8703199Z (EngineCore_DP0 pid=12438) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:42.8703720Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8704468Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:42.8705167Z (EngineCore_DP0 pid=12438) return _is_fa2_supported(device)[0] 2025-10-10T02:01:42.8705596Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8706319Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:42.8707034Z (EngineCore_DP0 pid=12438) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:42.8707493Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8708162Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:42.8708810Z (EngineCore_DP0 pid=12438) prop = get_device_properties(device) 2025-10-10T02:01:42.8709230Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:42.8709897Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:42.8710557Z (EngineCore_DP0 pid=12438) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:42.8710951Z (EngineCore_DP0 pid=12438) ^^^^^^^^^^^^ 2025-10-10T02:01:42.8711532Z (EngineCore_DP0 pid=12438) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:42.8712095Z (EngineCore_DP0 pid=12438) raise RuntimeError( 2025-10-10T02:01:42.8712782Z (EngineCore_DP0 pid=12438) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:43.2725429Z FAILED 2025-10-10T02:01:43.2853127Z models/test_initialization.py::test_can_initialize_large_subset[MiniCPMO] Fork a new process to run a test 12442 2025-10-10T02:01:43.2864636Z Fork a new process to run a test 0 2025-10-10T02:01:43.3137503Z INFO 10-10 02:01:43 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='MiniCPMO', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'openbmb/MiniCPM-o-2_6'} 2025-10-10T02:01:43.3970172Z 2025-10-10T02:01:43.3972269Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:43.3972784Z config.json: 3.44kB [00:00, 20.2MB/s] 2025-10-10T02:01:43.4586028Z 2025-10-10T02:01:43.4588533Z configuration_minicpm.py: 0.00B [00:00, ?B/s] 2025-10-10T02:01:43.4588916Z configuration_minicpm.py: 7.55kB [00:00, 34.0MB/s] 2025-10-10T02:01:43.5195876Z 2025-10-10T02:01:43.5200245Z modeling_navit_siglip.py: 0.00B [00:00, ?B/s] 2025-10-10T02:01:43.5200725Z modeling_navit_siglip.py: 42.1kB [00:00, 75.3MB/s] 2025-10-10T02:01:43.5308654Z A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6: 2025-10-10T02:01:43.5309339Z - modeling_navit_siglip.py 2025-10-10T02:01:43.5310102Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:01:43.5310959Z A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6: 2025-10-10T02:01:43.5311475Z - configuration_minicpm.py 2025-10-10T02:01:43.5311740Z - modeling_navit_siglip.py 2025-10-10T02:01:43.5312472Z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. 2025-10-10T02:01:43.8773157Z 2025-10-10T02:01:43.8773746Z preprocessor_config.json: 0% 0.00/714 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:52.2637458Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:52.2637911Z 2025-10-10T02:01:52.2638317Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:52.2638709Z 2025-10-10T02:01:52.2639527Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:52.2640130Z 2025-10-10T02:01:52.2640524Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:52.2640915Z 2025-10-10T02:01:52.2641263Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2641613Z 2025-10-10T02:01:52.2642205Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:52.2642976Z 2025-10-10T02:01:52.2643481Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:52.2643895Z 2025-10-10T02:01:52.2644526Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:52.2645077Z 2025-10-10T02:01:52.2645398Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:52.2645707Z 2025-10-10T02:01:52.2645984Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2646273Z 2025-10-10T02:01:52.2647186Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:52.2648329Z 2025-10-10T02:01:52.2648855Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] self._init_executor() 2025-10-10T02:01:52.2649531Z 2025-10-10T02:01:52.2650493Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:52.2651073Z 2025-10-10T02:01:52.2651426Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:52.2651865Z 2025-10-10T02:01:52.2652417Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:52.2652912Z 2025-10-10T02:01:52.2653265Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:52.2653599Z 2025-10-10T02:01:52.2653968Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2654261Z 2025-10-10T02:01:52.2654759Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:52.2655214Z 2025-10-10T02:01:52.2655485Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:52.2655760Z 2025-10-10T02:01:52.2656008Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2656270Z 2025-10-10T02:01:52.2656796Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:52.2657276Z 2025-10-10T02:01:52.2657561Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:52.2657851Z 2025-10-10T02:01:52.2658127Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2658394Z 2025-10-10T02:01:52.2658924Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:52.2659404Z 2025-10-10T02:01:52.2659709Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:52.2660018Z 2025-10-10T02:01:52.2660291Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2660570Z 2025-10-10T02:01:52.2661008Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:52.2661426Z 2025-10-10T02:01:52.2661761Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:52.2662100Z 2025-10-10T02:01:52.2662387Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2662675Z 2025-10-10T02:01:52.2663013Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:52.2663342Z 2025-10-10T02:01:52.2663696Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:52.2664084Z 2025-10-10T02:01:52.2664515Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:52.2664884Z 2025-10-10T02:01:52.2665234Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:52.2665566Z 2025-10-10T02:01:52.2665923Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:52.2666332Z 2025-10-10T02:01:52.2666708Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:52.2667073Z 2025-10-10T02:01:52.2667569Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:52.2668037Z 2025-10-10T02:01:52.2668435Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:52.2668777Z 2025-10-10T02:01:52.2669293Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:52.2669769Z 2025-10-10T02:01:52.2670146Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:52.2670524Z 2025-10-10T02:01:52.2671101Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:52.2671624Z 2025-10-10T02:01:52.2671929Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:52.2672229Z 2025-10-10T02:01:52.2672857Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:52.2673427Z 2025-10-10T02:01:52.2673793Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:52.2674155Z 2025-10-10T02:01:52.2674408Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2674680Z 2025-10-10T02:01:52.2675250Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:52.2675771Z 2025-10-10T02:01:52.2676080Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:52.2676385Z 2025-10-10T02:01:52.2676656Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2676926Z 2025-10-10T02:01:52.2677529Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:52.2678076Z 2025-10-10T02:01:52.2678360Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:52.2678640Z 2025-10-10T02:01:52.2678924Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2679301Z 2025-10-10T02:01:52.2679954Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:52.2680554Z 2025-10-10T02:01:52.2680877Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:52.2681193Z 2025-10-10T02:01:52.2681548Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2681830Z 2025-10-10T02:01:52.2682366Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:52.2682856Z 2025-10-10T02:01:52.2683140Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:52.2683428Z 2025-10-10T02:01:52.2683700Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2684023Z 2025-10-10T02:01:52.2684551Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:52.2685031Z 2025-10-10T02:01:52.2685350Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:52.2685655Z 2025-10-10T02:01:52.2685881Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:52.2686130Z 2025-10-10T02:01:52.2686610Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:52.2687069Z 2025-10-10T02:01:52.2687316Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] raise RuntimeError( 2025-10-10T02:01:52.2687579Z 2025-10-10T02:01:52.2688144Z (EngineCore_DP0 pid=12500) ERROR 10-10 02:01:52 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:52.2688837Z (EngineCore_DP0 pid=12500) Process EngineCore_DP0: 2025-10-10T02:01:52.2689273Z (EngineCore_DP0 pid=12500) Traceback (most recent call last): 2025-10-10T02:01:52.2689880Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:52.2690395Z (EngineCore_DP0 pid=12500) self.run() 2025-10-10T02:01:52.2690915Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:52.2691484Z (EngineCore_DP0 pid=12500) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:52.2692152Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:52.2692715Z (EngineCore_DP0 pid=12500) raise e 2025-10-10T02:01:52.2693304Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:52.2693935Z (EngineCore_DP0 pid=12500) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:52.2694377Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2695002Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:52.2695701Z (EngineCore_DP0 pid=12500) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:52.2696846Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:52.2697508Z (EngineCore_DP0 pid=12500) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:52.2697972Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2698743Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:52.2699345Z (EngineCore_DP0 pid=12500) self._init_executor() 2025-10-10T02:01:52.2700015Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:52.2700704Z (EngineCore_DP0 pid=12500) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:52.2701499Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:52.2702220Z (EngineCore_DP0 pid=12500) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:52.2702722Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2703366Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:52.2703957Z (EngineCore_DP0 pid=12500) return func(*args, **kwargs) 2025-10-10T02:01:52.2704335Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2704974Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:52.2705609Z (EngineCore_DP0 pid=12500) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:52.2706036Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2706702Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:52.2707368Z (EngineCore_DP0 pid=12500) module = importlib.import_module(module_name) 2025-10-10T02:01:52.2707817Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2708399Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:52.2709023Z (EngineCore_DP0 pid=12500) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:52.2709515Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2710007Z (EngineCore_DP0 pid=12500) File "", line 1387, in _gcd_import 2025-10-10T02:01:52.2710560Z (EngineCore_DP0 pid=12500) File "", line 1360, in _find_and_load 2025-10-10T02:01:52.2711142Z (EngineCore_DP0 pid=12500) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:52.2711739Z (EngineCore_DP0 pid=12500) File "", line 935, in _load_unlocked 2025-10-10T02:01:52.2712315Z (EngineCore_DP0 pid=12500) File "", line 999, in exec_module 2025-10-10T02:01:52.2713003Z (EngineCore_DP0 pid=12500) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:52.2713940Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:52.2714757Z (EngineCore_DP0 pid=12500) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:52.2715492Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:52.2716266Z (EngineCore_DP0 pid=12500) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:52.2717057Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:52.2717731Z (EngineCore_DP0 pid=12500) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:52.2718523Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:52.2719464Z (EngineCore_DP0 pid=12500) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:52.2719952Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2720645Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:52.2721331Z (EngineCore_DP0 pid=12500) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:52.2721769Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2722506Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:52.2723206Z (EngineCore_DP0 pid=12500) return _is_fa2_supported(device)[0] 2025-10-10T02:01:52.2723619Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2724369Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:52.2725116Z (EngineCore_DP0 pid=12500) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:52.2725572Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2726248Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:52.2726887Z (EngineCore_DP0 pid=12500) prop = get_device_properties(device) 2025-10-10T02:01:52.2727305Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:52.2727967Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:52.2728628Z (EngineCore_DP0 pid=12500) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:52.2729028Z (EngineCore_DP0 pid=12500) ^^^^^^^^^^^^ 2025-10-10T02:01:52.2729613Z (EngineCore_DP0 pid=12500) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:52.2730196Z (EngineCore_DP0 pid=12500) raise RuntimeError( 2025-10-10T02:01:52.2730884Z (EngineCore_DP0 pid=12500) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:52.6786396Z FAILED 2025-10-10T02:01:52.6916796Z models/test_initialization.py::test_can_initialize_large_subset[DeepseekV3ForCausalLM] Fork a new process to run a test 12504 2025-10-10T02:01:52.6927313Z Fork a new process to run a test 0 2025-10-10T02:01:52.7198279Z INFO 10-10 02:01:52 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='DeepseekV3ForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'deepseek-ai/DeepSeek-V3'} 2025-10-10T02:01:52.8465491Z 2025-10-10T02:01:52.8468309Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:52.8468651Z config.json: 1.66kB [00:00, 4.29MB/s] 2025-10-10T02:01:52.8963237Z INFO 10-10 02:01:52 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T02:01:52.9829704Z INFO 10-10 02:01:52 [model.py:551] Resolved architecture: DeepseekV3ForCausalLM 2025-10-10T02:01:52.9830189Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:01:53.0086073Z INFO 10-10 02:01:53 [model.py:1545] Using max model len 163840 2025-10-10T02:01:53.1101758Z The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. 2025-10-10T02:01:53.2782709Z INFO 10-10 02:01:53 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:01:53.3149740Z 2025-10-10T02:01:53.3151474Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:53.3151966Z tokenizer_config.json: 3.13kB [00:00, 23.0MB/s] 2025-10-10T02:01:53.4331432Z 2025-10-10T02:01:53.4591526Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T02:01:53.4591984Z tokenizer.json: 7.85MB [00:00, 302MB/s] 2025-10-10T02:01:54.0330045Z INFO 10-10 02:01:54 [config.py:388] Replacing legacy 'type' key with 'rope_type' 2025-10-10T02:01:54.0532404Z (EngineCore_DP0 pid=12512) INFO 10-10 02:01:54 [core.py:648] Waiting for init message from front-end. 2025-10-10T02:01:54.0611552Z (EngineCore_DP0 pid=12512) INFO 10-10 02:01:54 [core.py:78] Initializing a V1 LLM engine (v0.11.0rc2.dev157+g0ad9951c4.d20251010) with config: model='deepseek-ai/DeepSeek-V3', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-V3', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=163840, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=deepseek-ai/DeepSeek-V3, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': ['+quant_fp8'], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': , 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:01:54.1927898Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] EngineCore failed to start. 2025-10-10T02:01:54.1928446Z 2025-10-10T02:01:54.1928852Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] Traceback (most recent call last): 2025-10-10T02:01:54.1929232Z 2025-10-10T02:01:54.1930207Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:54.1930808Z 2025-10-10T02:01:54.1931211Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:54.1931594Z 2025-10-10T02:01:54.1931964Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1932335Z 2025-10-10T02:01:54.1933051Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:54.1933971Z 2025-10-10T02:01:54.1934441Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:54.1934866Z 2025-10-10T02:01:54.1935459Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:54.1936019Z 2025-10-10T02:01:54.1936412Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:54.1936822Z 2025-10-10T02:01:54.1937247Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1937674Z 2025-10-10T02:01:54.1938602Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:54.1939371Z 2025-10-10T02:01:54.1939899Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] self._init_executor() 2025-10-10T02:01:54.1940234Z 2025-10-10T02:01:54.1941027Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:54.1941573Z 2025-10-10T02:01:54.1941924Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:54.1942252Z 2025-10-10T02:01:54.1942801Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:54.1943309Z 2025-10-10T02:01:54.1943657Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:54.1944005Z 2025-10-10T02:01:54.1944298Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1944587Z 2025-10-10T02:01:54.1945082Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:54.1945540Z 2025-10-10T02:01:54.1945806Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:01:54.1946074Z 2025-10-10T02:01:54.1946537Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1946902Z 2025-10-10T02:01:54.1947460Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:54.1947955Z 2025-10-10T02:01:54.1948256Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:54.1948600Z 2025-10-10T02:01:54.1948875Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1949151Z 2025-10-10T02:01:54.1949684Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:54.1950208Z 2025-10-10T02:01:54.1950518Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:01:54.1950837Z 2025-10-10T02:01:54.1951179Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1951460Z 2025-10-10T02:01:54.1951937Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:54.1952383Z 2025-10-10T02:01:54.1952739Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:54.1953074Z 2025-10-10T02:01:54.1953369Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1953646Z 2025-10-10T02:01:54.1953992Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:01:54.1954329Z 2025-10-10T02:01:54.1954683Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:01:54.1955027Z 2025-10-10T02:01:54.1955404Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:54.1955773Z 2025-10-10T02:01:54.1956118Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:01:54.1956465Z 2025-10-10T02:01:54.1956827Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:01:54.1957180Z 2025-10-10T02:01:54.1957560Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:54.1957920Z 2025-10-10T02:01:54.1958430Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:54.1958897Z 2025-10-10T02:01:54.1959404Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:54.1959744Z 2025-10-10T02:01:54.1960273Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:54.1960755Z 2025-10-10T02:01:54.1961178Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:54.1961596Z 2025-10-10T02:01:54.1962155Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:54.1962668Z 2025-10-10T02:01:54.1962969Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:01:54.1963271Z 2025-10-10T02:01:54.1963967Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:54.1964550Z 2025-10-10T02:01:54.1964917Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:54.1965269Z 2025-10-10T02:01:54.1965536Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1965802Z 2025-10-10T02:01:54.1966408Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:54.1966932Z 2025-10-10T02:01:54.1967235Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:01:54.1967531Z 2025-10-10T02:01:54.1967807Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1968085Z 2025-10-10T02:01:54.1968686Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:54.1969242Z 2025-10-10T02:01:54.1969528Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:01:54.1969821Z 2025-10-10T02:01:54.1970085Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1970360Z 2025-10-10T02:01:54.1970935Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:54.1971465Z 2025-10-10T02:01:54.1971789Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:54.1972114Z 2025-10-10T02:01:54.1972395Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1972669Z 2025-10-10T02:01:54.1973199Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:54.1973683Z 2025-10-10T02:01:54.1973974Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:01:54.1974262Z 2025-10-10T02:01:54.1974588Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1974987Z 2025-10-10T02:01:54.1975528Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:54.1976019Z 2025-10-10T02:01:54.1976336Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:01:54.1976653Z 2025-10-10T02:01:54.1976937Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:01:54.1977218Z 2025-10-10T02:01:54.1977720Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:54.1978183Z 2025-10-10T02:01:54.1978436Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] raise RuntimeError( 2025-10-10T02:01:54.1978733Z 2025-10-10T02:01:54.1979306Z (EngineCore_DP0 pid=12512) ERROR 10-10 02:01:54 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:54.1980003Z (EngineCore_DP0 pid=12512) Process EngineCore_DP0: 2025-10-10T02:01:54.1980430Z (EngineCore_DP0 pid=12512) Traceback (most recent call last): 2025-10-10T02:01:54.1981053Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:01:54.1981575Z (EngineCore_DP0 pid=12512) self.run() 2025-10-10T02:01:54.1982146Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:01:54.1982732Z (EngineCore_DP0 pid=12512) self._target(*self._args, **self._kwargs) 2025-10-10T02:01:54.1983406Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:01:54.1983976Z (EngineCore_DP0 pid=12512) raise e 2025-10-10T02:01:54.1984568Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:01:54.1985211Z (EngineCore_DP0 pid=12512) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:01:54.1985663Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1986305Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:01:54.1986945Z (EngineCore_DP0 pid=12512) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:01:54.1987613Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:01:54.1988239Z (EngineCore_DP0 pid=12512) self.model_executor = executor_class(vllm_config) 2025-10-10T02:01:54.1988698Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1989371Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:01:54.1989959Z (EngineCore_DP0 pid=12512) self._init_executor() 2025-10-10T02:01:54.1990628Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:01:54.1991320Z (EngineCore_DP0 pid=12512) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:01:54.1992052Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:01:54.1992774Z (EngineCore_DP0 pid=12512) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:01:54.1993270Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1993973Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:01:54.1994610Z (EngineCore_DP0 pid=12512) return func(*args, **kwargs) 2025-10-10T02:01:54.1995004Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1995649Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:01:54.1996489Z (EngineCore_DP0 pid=12512) worker_class = resolve_obj_by_qualname( 2025-10-10T02:01:54.1997039Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1997723Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:01:54.1998399Z (EngineCore_DP0 pid=12512) module = importlib.import_module(module_name) 2025-10-10T02:01:54.1998857Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.1999603Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:01:54.2000259Z (EngineCore_DP0 pid=12512) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:01:54.2000758Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.2001265Z (EngineCore_DP0 pid=12512) File "", line 1387, in _gcd_import 2025-10-10T02:01:54.2001825Z (EngineCore_DP0 pid=12512) File "", line 1360, in _find_and_load 2025-10-10T02:01:54.2002413Z (EngineCore_DP0 pid=12512) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:01:54.2002992Z (EngineCore_DP0 pid=12512) File "", line 935, in _load_unlocked 2025-10-10T02:01:54.2003572Z (EngineCore_DP0 pid=12512) File "", line 999, in exec_module 2025-10-10T02:01:54.2004181Z (EngineCore_DP0 pid=12512) File "", line 488, in _call_with_frames_removed 2025-10-10T02:01:54.2004920Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:01:54.2005607Z (EngineCore_DP0 pid=12512) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:01:54.2006329Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:01:54.2007057Z (EngineCore_DP0 pid=12512) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:01:54.2007841Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:01:54.2008521Z (EngineCore_DP0 pid=12512) class FlashAttentionMetadataBuilder( 2025-10-10T02:01:54.2009328Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:01:54.2010145Z (EngineCore_DP0 pid=12512) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:01:54.2010624Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.2011319Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:01:54.2012079Z (EngineCore_DP0 pid=12512) if not is_fa_version_supported(fa_version): 2025-10-10T02:01:54.2012589Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.2013369Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:01:54.2014088Z (EngineCore_DP0 pid=12512) return _is_fa2_supported(device)[0] 2025-10-10T02:01:54.2014507Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.2015276Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:01:54.2016008Z (EngineCore_DP0 pid=12512) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:01:54.2016488Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.2017173Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:01:54.2017861Z (EngineCore_DP0 pid=12512) prop = get_device_properties(device) 2025-10-10T02:01:54.2018284Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:01:54.2018952Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:01:54.2019626Z (EngineCore_DP0 pid=12512) _lazy_init() # will define _get_device_properties 2025-10-10T02:01:54.2020028Z (EngineCore_DP0 pid=12512) ^^^^^^^^^^^^ 2025-10-10T02:01:54.2020609Z (EngineCore_DP0 pid=12512) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:01:54.2021172Z (EngineCore_DP0 pid=12512) raise RuntimeError( 2025-10-10T02:01:54.2021865Z (EngineCore_DP0 pid=12512) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:01:54.6034771Z FAILED 2025-10-10T02:01:54.6162770Z models/test_initialization.py::test_can_initialize_large_subset[GteModel] Fork a new process to run a test 12516 2025-10-10T02:01:54.6174686Z Fork a new process to run a test 0 2025-10-10T02:01:54.6448896Z INFO 10-10 02:01:54 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GteModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Snowflake/snowflake-arctic-embed-m-v2.0'} 2025-10-10T02:01:54.7636109Z 2025-10-10T02:01:54.7639414Z config.json: 0% 0.00/971 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:03.4418104Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:03.4418535Z 2025-10-10T02:02:03.4418968Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:03.4419277Z 2025-10-10T02:02:03.4419957Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:03.4420456Z 2025-10-10T02:02:03.4421015Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:03.4421336Z 2025-10-10T02:02:03.4421637Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4421935Z 2025-10-10T02:02:03.4422418Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:03.4422866Z 2025-10-10T02:02:03.4423199Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:03.4423523Z 2025-10-10T02:02:03.4423995Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:03.4424464Z 2025-10-10T02:02:03.4424914Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:03.4425253Z 2025-10-10T02:02:03.4425579Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4425945Z 2025-10-10T02:02:03.4426494Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:03.4426979Z 2025-10-10T02:02:03.4427245Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] self._init_executor() 2025-10-10T02:02:03.4427520Z 2025-10-10T02:02:03.4428072Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:03.4428576Z 2025-10-10T02:02:03.4429010Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:03.4429347Z 2025-10-10T02:02:03.4430016Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:03.4430661Z 2025-10-10T02:02:03.4431151Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:03.4431725Z 2025-10-10T02:02:03.4432063Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4432548Z 2025-10-10T02:02:03.4433536Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:03.4434290Z 2025-10-10T02:02:03.4434701Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:03.4435004Z 2025-10-10T02:02:03.4435276Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4435665Z 2025-10-10T02:02:03.4436223Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:03.4436710Z 2025-10-10T02:02:03.4437010Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:03.4437303Z 2025-10-10T02:02:03.4437576Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4437855Z 2025-10-10T02:02:03.4438438Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:03.4438935Z 2025-10-10T02:02:03.4439364Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:03.4439684Z 2025-10-10T02:02:03.4439961Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4440238Z 2025-10-10T02:02:03.4440689Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:03.4441105Z 2025-10-10T02:02:03.4441458Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:03.4441795Z 2025-10-10T02:02:03.4442089Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4442369Z 2025-10-10T02:02:03.4442719Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:03.4443059Z 2025-10-10T02:02:03.4443412Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:03.4443754Z 2025-10-10T02:02:03.4444130Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:03.4444490Z 2025-10-10T02:02:03.4444840Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:03.4445177Z 2025-10-10T02:02:03.4445550Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:03.4445900Z 2025-10-10T02:02:03.4446277Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:03.4446637Z 2025-10-10T02:02:03.4447164Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:03.4447634Z 2025-10-10T02:02:03.4448045Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:03.4448393Z 2025-10-10T02:02:03.4448963Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:03.4449447Z 2025-10-10T02:02:03.4449832Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:03.4450248Z 2025-10-10T02:02:03.4450801Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:03.4451313Z 2025-10-10T02:02:03.4451607Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:03.4451917Z 2025-10-10T02:02:03.4452547Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:03.4453134Z 2025-10-10T02:02:03.4453543Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:03.4453902Z 2025-10-10T02:02:03.4454166Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4454431Z 2025-10-10T02:02:03.4455014Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:03.4455535Z 2025-10-10T02:02:03.4455839Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:03.4456134Z 2025-10-10T02:02:03.4456413Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4456690Z 2025-10-10T02:02:03.4457291Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:03.4457849Z 2025-10-10T02:02:03.4458131Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:03.4458424Z 2025-10-10T02:02:03.4458684Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4458957Z 2025-10-10T02:02:03.4459546Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:03.4460084Z 2025-10-10T02:02:03.4460405Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:03.4460717Z 2025-10-10T02:02:03.4460997Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4461276Z 2025-10-10T02:02:03.4461809Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:03.4462298Z 2025-10-10T02:02:03.4462589Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:03.4462882Z 2025-10-10T02:02:03.4463154Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4463439Z 2025-10-10T02:02:03.4464048Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:03.4464586Z 2025-10-10T02:02:03.4464907Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:03.4465237Z 2025-10-10T02:02:03.4465468Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:03.4465754Z 2025-10-10T02:02:03.4466257Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:03.4466713Z 2025-10-10T02:02:03.4466967Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] raise RuntimeError( 2025-10-10T02:02:03.4467230Z 2025-10-10T02:02:03.4467842Z (EngineCore_DP0 pid=12595) ERROR 10-10 02:02:03 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:03.4468578Z (EngineCore_DP0 pid=12595) Process EngineCore_DP0: 2025-10-10T02:02:03.4468989Z (EngineCore_DP0 pid=12595) Traceback (most recent call last): 2025-10-10T02:02:03.4469611Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:03.4470132Z (EngineCore_DP0 pid=12595) self.run() 2025-10-10T02:02:03.4470659Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:03.4471226Z (EngineCore_DP0 pid=12595) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:03.4471899Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:03.4472448Z (EngineCore_DP0 pid=12595) raise e 2025-10-10T02:02:03.4473042Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:03.4473684Z (EngineCore_DP0 pid=12595) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:03.4474136Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4474770Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:03.4475419Z (EngineCore_DP0 pid=12595) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:03.4476082Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:03.4476709Z (EngineCore_DP0 pid=12595) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:03.4477169Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4477826Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:03.4478411Z (EngineCore_DP0 pid=12595) self._init_executor() 2025-10-10T02:02:03.4479186Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:03.4479889Z (EngineCore_DP0 pid=12595) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:03.4480681Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:03.4481474Z (EngineCore_DP0 pid=12595) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:03.4481975Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4482637Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:03.4483274Z (EngineCore_DP0 pid=12595) return func(*args, **kwargs) 2025-10-10T02:02:03.4483663Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4484308Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:03.4484962Z (EngineCore_DP0 pid=12595) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:03.4485391Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4486109Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:03.4486777Z (EngineCore_DP0 pid=12595) module = importlib.import_module(module_name) 2025-10-10T02:02:03.4487229Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4487830Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:03.4488463Z (EngineCore_DP0 pid=12595) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:03.4488963Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4489466Z (EngineCore_DP0 pid=12595) File "", line 1387, in _gcd_import 2025-10-10T02:02:03.4490033Z (EngineCore_DP0 pid=12595) File "", line 1360, in _find_and_load 2025-10-10T02:02:03.4490622Z (EngineCore_DP0 pid=12595) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:03.4491203Z (EngineCore_DP0 pid=12595) File "", line 935, in _load_unlocked 2025-10-10T02:02:03.4491781Z (EngineCore_DP0 pid=12595) File "", line 999, in exec_module 2025-10-10T02:02:03.4492401Z (EngineCore_DP0 pid=12595) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:03.4493145Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:03.4493834Z (EngineCore_DP0 pid=12595) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:03.4494559Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:03.4495295Z (EngineCore_DP0 pid=12595) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:03.4496485Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:03.4497243Z (EngineCore_DP0 pid=12595) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:03.4498056Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:03.4499109Z (EngineCore_DP0 pid=12595) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:03.4499690Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4500424Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:03.4501120Z (EngineCore_DP0 pid=12595) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:03.4501677Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4509258Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:03.4510047Z (EngineCore_DP0 pid=12595) return _is_fa2_supported(device)[0] 2025-10-10T02:02:03.4510494Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4511262Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:03.4512122Z (EngineCore_DP0 pid=12595) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:03.4512628Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4513343Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:03.4514017Z (EngineCore_DP0 pid=12595) prop = get_device_properties(device) 2025-10-10T02:02:03.4514453Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:03.4515156Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:03.4515833Z (EngineCore_DP0 pid=12595) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:03.4516269Z (EngineCore_DP0 pid=12595) ^^^^^^^^^^^^ 2025-10-10T02:02:03.4516870Z (EngineCore_DP0 pid=12595) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:03.4517447Z (EngineCore_DP0 pid=12595) raise RuntimeError( 2025-10-10T02:02:03.4518150Z (EngineCore_DP0 pid=12595) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:03.8801664Z FAILED 2025-10-10T02:02:03.8929839Z models/test_initialization.py::test_can_initialize_large_subset[LlavaOnevisionForConditionalGeneration] Fork a new process to run a test 12599 2025-10-10T02:02:03.8942167Z Fork a new process to run a test 0 2025-10-10T02:02:03.9218864Z INFO 10-10 02:02:03 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlavaOnevisionForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'llava-hf/llava-onevision-qwen2-0.5b-ov-hf'} 2025-10-10T02:02:03.9997344Z 2025-10-10T02:02:03.9999549Z config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:02:03.9999900Z config.json: 2.59kB [00:00, 14.1MB/s] 2025-10-10T02:02:04.0970384Z 2025-10-10T02:02:04.0970779Z preprocessor_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:02:04.0971169Z preprocessor_config.json: 1.73kB [00:00, 23.0MB/s] 2025-10-10T02:02:11.2168776Z INFO 10-10 02:02:11 [model.py:551] Resolved architecture: LlavaOnevisionForConditionalGeneration 2025-10-10T02:02:11.2169351Z `torch_dtype` is deprecated! Use `dtype` instead! 2025-10-10T02:02:11.2416048Z INFO 10-10 02:02:11 [model.py:1545] Using max model len 32768 2025-10-10T02:02:11.4421617Z INFO 10-10 02:02:11 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192. 2025-10-10T02:02:11.4813443Z 2025-10-10T02:02:11.4814860Z tokenizer_config.json: 0.00B [00:00, ?B/s] 2025-10-10T02:02:11.4815257Z tokenizer_config.json: 1.80kB [00:00, 8.97MB/s] 2025-10-10T02:02:11.5430735Z 2025-10-10T02:02:11.5584455Z vocab.json: 0.00B [00:00, ?B/s] 2025-10-10T02:02:11.5584888Z vocab.json: 2.78MB [00:00, 179MB/s] 2025-10-10T02:02:11.6074641Z 2025-10-10T02:02:11.6151705Z merges.txt: 0.00B [00:00, ?B/s] 2025-10-10T02:02:11.6152125Z merges.txt: 1.67MB [00:00, 216MB/s] 2025-10-10T02:02:11.6525654Z 2025-10-10T02:02:11.6761089Z tokenizer.json: 0.00B [00:00, ?B/s] 2025-10-10T02:02:11.6761537Z tokenizer.json: 7.03MB [00:00, 299MB/s] 2025-10-10T02:02:11.7261259Z 2025-10-10T02:02:11.7266065Z added_tokens.json: 0% 0.00/122 [00:00", line 1387, in _gcd_import 2025-10-10T02:02:12.3126816Z 2025-10-10T02:02:12.3127018Z ERROR 10-10 02:02:12 [registry.py:542] File "", line 1360, in _find_and_load 2025-10-10T02:02:12.3127303Z 2025-10-10T02:02:12.3127530Z ERROR 10-10 02:02:12 [registry.py:542] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:12.3127840Z 2025-10-10T02:02:12.3128038Z ERROR 10-10 02:02:12 [registry.py:542] File "", line 935, in _load_unlocked 2025-10-10T02:02:12.3128310Z 2025-10-10T02:02:12.3128533Z ERROR 10-10 02:02:12 [registry.py:542] File "", line 999, in exec_module 2025-10-10T02:02:12.3128822Z 2025-10-10T02:02:12.3129312Z ERROR 10-10 02:02:12 [registry.py:542] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:12.3129734Z 2025-10-10T02:02:12.3130139Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava_onevision.py", line 28, in 2025-10-10T02:02:12.3130593Z 2025-10-10T02:02:12.3130823Z ERROR 10-10 02:02:12 [registry.py:542] from .llava import LlavaDummyInputsBuilder, init_vision_tower_for_llava 2025-10-10T02:02:12.3131206Z 2025-10-10T02:02:12.3131564Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/llava.py", line 40, in 2025-10-10T02:02:12.3131989Z 2025-10-10T02:02:12.3132199Z ERROR 10-10 02:02:12 [registry.py:542] from .pixtral import PixtralHFEncoderInfo, PixtralHFVisionModel 2025-10-10T02:02:12.3132491Z 2025-10-10T02:02:12.3132847Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 58, in 2025-10-10T02:02:12.3133289Z 2025-10-10T02:02:12.3133515Z ERROR 10-10 02:02:12 [registry.py:542] from xformers import ops as xops 2025-10-10T02:02:12.3133738Z 2025-10-10T02:02:12.3134055Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py", line 9, in 2025-10-10T02:02:12.3134442Z 2025-10-10T02:02:12.3134566Z ERROR 10-10 02:02:12 [registry.py:542] from .fmha import ( 2025-10-10T02:02:12.3134756Z 2025-10-10T02:02:12.3135088Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py", line 10, in 2025-10-10T02:02:12.3135496Z 2025-10-10T02:02:12.3135725Z ERROR 10-10 02:02:12 [registry.py:542] from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T02:02:12.3136027Z 2025-10-10T02:02:12.3136381Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py", line 124, in 2025-10-10T02:02:12.3136816Z 2025-10-10T02:02:12.3136976Z ERROR 10-10 02:02:12 [registry.py:542] if TYPE_CHECKING or _is_triton_available(): 2025-10-10T02:02:12.3137209Z 2025-10-10T02:02:12.3137367Z ERROR 10-10 02:02:12 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:12.3137581Z 2025-10-10T02:02:12.3137906Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 38, in func_wrapper 2025-10-10T02:02:12.3138293Z 2025-10-10T02:02:12.3138406Z ERROR 10-10 02:02:12 [registry.py:542] value = func() 2025-10-10T02:02:12.3138586Z 2025-10-10T02:02:12.3138688Z ERROR 10-10 02:02:12 [registry.py:542] ^^^^^^ 2025-10-10T02:02:12.3138869Z 2025-10-10T02:02:12.3139207Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py", line 54, in _is_triton_available 2025-10-10T02:02:12.3139626Z 2025-10-10T02:02:12.3139810Z ERROR 10-10 02:02:12 [registry.py:542] if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T02:02:12.3140071Z 2025-10-10T02:02:12.3140207Z ERROR 10-10 02:02:12 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:12.3140419Z 2025-10-10T02:02:12.3140773Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:12.3141198Z 2025-10-10T02:02:12.3141358Z ERROR 10-10 02:02:12 [registry.py:542] prop = get_device_properties(device) 2025-10-10T02:02:12.3141579Z 2025-10-10T02:02:12.3141706Z ERROR 10-10 02:02:12 [registry.py:542] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:12.3141916Z 2025-10-10T02:02:12.3142318Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:12.3142796Z 2025-10-10T02:02:12.3142977Z ERROR 10-10 02:02:12 [registry.py:542] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:12.3143237Z 2025-10-10T02:02:12.3143340Z ERROR 10-10 02:02:12 [registry.py:542] ^^^^^^^^^^^^ 2025-10-10T02:02:12.3143511Z 2025-10-10T02:02:12.3143832Z ERROR 10-10 02:02:12 [registry.py:542] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:12.3144259Z 2025-10-10T02:02:12.3144380Z ERROR 10-10 02:02:12 [registry.py:542] raise RuntimeError( 2025-10-10T02:02:12.3144566Z 2025-10-10T02:02:12.3144961Z ERROR 10-10 02:02:12 [registry.py:542] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:12.6976686Z FAILED 2025-10-10T02:02:12.7105989Z models/test_initialization.py::test_can_initialize_large_subset[Qwen3VLMoeForConditionalGeneration] Fork a new process to run a test 12655 2025-10-10T02:02:12.7116405Z Fork a new process to run a test 0 2025-10-10T02:02:12.7118543Z Model is not available online 2025-10-10T02:02:13.0172472Z PASSED 2025-10-10T02:02:13.0300923Z models/test_initialization.py::test_can_initialize_large_subset[QWenLMHeadModel] Fork a new process to run a test 12656 2025-10-10T02:02:13.0312721Z Fork a new process to run a test 0 2025-10-10T02:02:13.0316515Z `transformers==4.56.2` installed, but `transformers<=4.53` is required to run this model. Reason: HF model uses remote code that is not compatible with latest Transformers 2025-10-10T02:02:13.3357366Z PASSED 2025-10-10T02:02:13.3561173Z models/test_initialization.py::test_can_initialize_large_subset[FuyuForCausalLM] Fork a new process to run a test 12657 2025-10-10T02:02:13.3571626Z Fork a new process to run a test 0 2025-10-10T02:02:13.3847202Z INFO 10-10 02:02:13 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='FuyuForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'adept/fuyu-8b'} 2025-10-10T02:02:13.4874428Z 2025-10-10T02:02:13.4875544Z config.json: 0% 0.00/768 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:23.1486465Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:23.1486870Z 2025-10-10T02:02:23.1487256Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:23.1487634Z 2025-10-10T02:02:23.1488351Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:23.1489010Z 2025-10-10T02:02:23.1489403Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:23.1489786Z 2025-10-10T02:02:23.1490134Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1490494Z 2025-10-10T02:02:23.1491137Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:23.1491723Z 2025-10-10T02:02:23.1492149Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:23.1492763Z 2025-10-10T02:02:23.1493489Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:23.1493955Z 2025-10-10T02:02:23.1494276Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:23.1494597Z 2025-10-10T02:02:23.1494887Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1495246Z 2025-10-10T02:02:23.1495761Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:23.1496439Z 2025-10-10T02:02:23.1496715Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] self._init_executor() 2025-10-10T02:02:23.1496995Z 2025-10-10T02:02:23.1497659Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:23.1498184Z 2025-10-10T02:02:23.1498519Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:23.1498868Z 2025-10-10T02:02:23.1499427Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:23.1499942Z 2025-10-10T02:02:23.1500289Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:23.1500628Z 2025-10-10T02:02:23.1501049Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1501461Z 2025-10-10T02:02:23.1502432Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:23.1503085Z 2025-10-10T02:02:23.1503558Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:23.1503857Z 2025-10-10T02:02:23.1504135Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1504399Z 2025-10-10T02:02:23.1504925Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:23.1505411Z 2025-10-10T02:02:23.1505712Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:23.1506013Z 2025-10-10T02:02:23.1506285Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1506564Z 2025-10-10T02:02:23.1507097Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:23.1507595Z 2025-10-10T02:02:23.1507897Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:23.1508194Z 2025-10-10T02:02:23.1508475Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1508747Z 2025-10-10T02:02:23.1509294Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:23.1509785Z 2025-10-10T02:02:23.1510145Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:23.1510487Z 2025-10-10T02:02:23.1510779Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1511123Z 2025-10-10T02:02:23.1511471Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:23.1511805Z 2025-10-10T02:02:23.1512155Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:23.1512500Z 2025-10-10T02:02:23.1512877Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:23.1513241Z 2025-10-10T02:02:23.1513622Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:23.1513955Z 2025-10-10T02:02:23.1514329Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:23.1514681Z 2025-10-10T02:02:23.1515063Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:23.1515421Z 2025-10-10T02:02:23.1515934Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:23.1516396Z 2025-10-10T02:02:23.1516749Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:23.1517085Z 2025-10-10T02:02:23.1517602Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:23.1518106Z 2025-10-10T02:02:23.1518489Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:23.1518857Z 2025-10-10T02:02:23.1519564Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:23.1520087Z 2025-10-10T02:02:23.1520383Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:23.1520681Z 2025-10-10T02:02:23.1521335Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:23.1521908Z 2025-10-10T02:02:23.1522283Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:23.1522635Z 2025-10-10T02:02:23.1522902Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1523168Z 2025-10-10T02:02:23.1523745Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:23.1524266Z 2025-10-10T02:02:23.1524613Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:23.1524985Z 2025-10-10T02:02:23.1525265Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1525561Z 2025-10-10T02:02:23.1526173Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:23.1526785Z 2025-10-10T02:02:23.1527071Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:23.1527362Z 2025-10-10T02:02:23.1527623Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1527890Z 2025-10-10T02:02:23.1528475Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:23.1529003Z 2025-10-10T02:02:23.1529363Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:23.1529679Z 2025-10-10T02:02:23.1529960Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1530232Z 2025-10-10T02:02:23.1530758Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:23.1531248Z 2025-10-10T02:02:23.1531528Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:23.1531824Z 2025-10-10T02:02:23.1532094Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1532376Z 2025-10-10T02:02:23.1532907Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:23.1533396Z 2025-10-10T02:02:23.1533709Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:23.1534017Z 2025-10-10T02:02:23.1534256Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:23.1534497Z 2025-10-10T02:02:23.1534994Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:23.1535447Z 2025-10-10T02:02:23.1535704Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] raise RuntimeError( 2025-10-10T02:02:23.1535967Z 2025-10-10T02:02:23.1536532Z (EngineCore_DP0 pid=12736) ERROR 10-10 02:02:23 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:23.1537237Z (EngineCore_DP0 pid=12736) Process EngineCore_DP0: 2025-10-10T02:02:23.1537660Z (EngineCore_DP0 pid=12736) Traceback (most recent call last): 2025-10-10T02:02:23.1538281Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:23.1538804Z (EngineCore_DP0 pid=12736) self.run() 2025-10-10T02:02:23.1539347Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:23.1539969Z (EngineCore_DP0 pid=12736) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:23.1540636Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:23.1541227Z (EngineCore_DP0 pid=12736) raise e 2025-10-10T02:02:23.1541815Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:23.1542448Z (EngineCore_DP0 pid=12736) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:23.1542936Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1543554Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:23.1544195Z (EngineCore_DP0 pid=12736) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:23.1544854Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:23.1546362Z (EngineCore_DP0 pid=12736) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:23.1546843Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1547487Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:23.1548079Z (EngineCore_DP0 pid=12736) self._init_executor() 2025-10-10T02:02:23.1548746Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:23.1549433Z (EngineCore_DP0 pid=12736) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:23.1550158Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:23.1550868Z (EngineCore_DP0 pid=12736) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:23.1551363Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1552015Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:23.1552606Z (EngineCore_DP0 pid=12736) return func(*args, **kwargs) 2025-10-10T02:02:23.1552990Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1553629Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:23.1554252Z (EngineCore_DP0 pid=12736) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:23.1554684Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1555357Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:23.1556012Z (EngineCore_DP0 pid=12736) module = importlib.import_module(module_name) 2025-10-10T02:02:23.1556460Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1557051Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:23.1557674Z (EngineCore_DP0 pid=12736) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:23.1558215Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1558755Z (EngineCore_DP0 pid=12736) File "", line 1387, in _gcd_import 2025-10-10T02:02:23.1559411Z (EngineCore_DP0 pid=12736) File "", line 1360, in _find_and_load 2025-10-10T02:02:23.1559996Z (EngineCore_DP0 pid=12736) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:23.1560627Z (EngineCore_DP0 pid=12736) File "", line 935, in _load_unlocked 2025-10-10T02:02:23.1561204Z (EngineCore_DP0 pid=12736) File "", line 999, in exec_module 2025-10-10T02:02:23.1561808Z (EngineCore_DP0 pid=12736) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:23.1562548Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:23.1563227Z (EngineCore_DP0 pid=12736) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:23.1563990Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:23.1564715Z (EngineCore_DP0 pid=12736) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:23.1565499Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:23.1566173Z (EngineCore_DP0 pid=12736) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:23.1566962Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:23.1567772Z (EngineCore_DP0 pid=12736) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:23.1568258Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1568952Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:23.1569647Z (EngineCore_DP0 pid=12736) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:23.1570095Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1570838Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:23.1571535Z (EngineCore_DP0 pid=12736) return _is_fa2_supported(device)[0] 2025-10-10T02:02:23.1571949Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1572663Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:23.1573381Z (EngineCore_DP0 pid=12736) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:23.1573828Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1574511Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:23.1575143Z (EngineCore_DP0 pid=12736) prop = get_device_properties(device) 2025-10-10T02:02:23.1575562Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:23.1576266Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:23.1576967Z (EngineCore_DP0 pid=12736) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:23.1577355Z (EngineCore_DP0 pid=12736) ^^^^^^^^^^^^ 2025-10-10T02:02:23.1577927Z (EngineCore_DP0 pid=12736) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:23.1578526Z (EngineCore_DP0 pid=12736) raise RuntimeError( 2025-10-10T02:02:23.1579210Z (EngineCore_DP0 pid=12736) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:23.6338544Z FAILED 2025-10-10T02:02:23.6467342Z models/test_initialization.py::test_can_initialize_large_subset[AquilaModel] Fork a new process to run a test 12740 2025-10-10T02:02:23.6479391Z Fork a new process to run a test 0 2025-10-10T02:02:23.6752074Z INFO 10-10 02:02:23 [utils.py:233] non-default args: {'trust_remote_code': True, 'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='AquilaModel', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/AquilaChat-7B'} 2025-10-10T02:02:23.9466904Z 2025-10-10T02:02:23.9467989Z config.json: 0% 0.00/707 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:25.2204382Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:25.2204976Z 2025-10-10T02:02:25.2205520Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:25.2206060Z 2025-10-10T02:02:25.2207047Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:25.2207919Z 2025-10-10T02:02:25.2208481Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:25.2209060Z 2025-10-10T02:02:25.2209588Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2210104Z 2025-10-10T02:02:25.2211063Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:25.2211941Z 2025-10-10T02:02:25.2212652Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:25.2213303Z 2025-10-10T02:02:25.2214295Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:25.2215174Z 2025-10-10T02:02:25.2215818Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:25.2216445Z 2025-10-10T02:02:25.2217021Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2217577Z 2025-10-10T02:02:25.2218562Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:25.2219453Z 2025-10-10T02:02:25.2220194Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] self._init_executor() 2025-10-10T02:02:25.2220852Z 2025-10-10T02:02:25.2221889Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:25.2222726Z 2025-10-10T02:02:25.2223305Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:25.2224016Z 2025-10-10T02:02:25.2224958Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:25.2225596Z 2025-10-10T02:02:25.2225995Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:25.2226368Z 2025-10-10T02:02:25.2226832Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2227143Z 2025-10-10T02:02:25.2227661Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:25.2228127Z 2025-10-10T02:02:25.2228406Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:25.2228693Z 2025-10-10T02:02:25.2228947Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2229225Z 2025-10-10T02:02:25.2229764Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:25.2230247Z 2025-10-10T02:02:25.2230545Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:25.2230838Z 2025-10-10T02:02:25.2231110Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2231379Z 2025-10-10T02:02:25.2231913Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:25.2232406Z 2025-10-10T02:02:25.2232710Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:25.2233015Z 2025-10-10T02:02:25.2233288Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2233566Z 2025-10-10T02:02:25.2234007Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:25.2234447Z 2025-10-10T02:02:25.2234798Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:25.2235133Z 2025-10-10T02:02:25.2235421Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2235701Z 2025-10-10T02:02:25.2236046Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:25.2236374Z 2025-10-10T02:02:25.2236727Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:25.2237130Z 2025-10-10T02:02:25.2237578Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:25.2237943Z 2025-10-10T02:02:25.2238286Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:25.2238623Z 2025-10-10T02:02:25.2238982Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:25.2239548Z 2025-10-10T02:02:25.2239933Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:25.2240300Z 2025-10-10T02:02:25.2240803Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:25.2241272Z 2025-10-10T02:02:25.2241673Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:25.2242032Z 2025-10-10T02:02:25.2242557Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:25.2243045Z 2025-10-10T02:02:25.2243416Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:25.2243779Z 2025-10-10T02:02:25.2244324Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:25.2244849Z 2025-10-10T02:02:25.2245156Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:25.2245455Z 2025-10-10T02:02:25.2246093Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:25.2246665Z 2025-10-10T02:02:25.2247038Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:25.2247390Z 2025-10-10T02:02:25.2247643Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2247921Z 2025-10-10T02:02:25.2248494Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:25.2249032Z 2025-10-10T02:02:25.2249333Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:25.2249636Z 2025-10-10T02:02:25.2249912Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2250190Z 2025-10-10T02:02:25.2250789Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:25.2251341Z 2025-10-10T02:02:25.2251630Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:25.2251912Z 2025-10-10T02:02:25.2252180Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2252447Z 2025-10-10T02:02:25.2253081Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:25.2253658Z 2025-10-10T02:02:25.2253987Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:25.2254296Z 2025-10-10T02:02:25.2254569Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2254889Z 2025-10-10T02:02:25.2255414Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:25.2255900Z 2025-10-10T02:02:25.2256185Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:25.2256479Z 2025-10-10T02:02:25.2256747Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2257062Z 2025-10-10T02:02:25.2257596Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:25.2258084Z 2025-10-10T02:02:25.2258403Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:25.2258708Z 2025-10-10T02:02:25.2258939Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:25.2259181Z 2025-10-10T02:02:25.2259661Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:25.2260122Z 2025-10-10T02:02:25.2260374Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] raise RuntimeError( 2025-10-10T02:02:25.2260635Z 2025-10-10T02:02:25.2261204Z (EngineCore_DP0 pid=12748) ERROR 10-10 02:02:25 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:25.2261898Z (EngineCore_DP0 pid=12748) Process EngineCore_DP0: 2025-10-10T02:02:25.2262322Z (EngineCore_DP0 pid=12748) Traceback (most recent call last): 2025-10-10T02:02:25.2262934Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:25.2263453Z (EngineCore_DP0 pid=12748) self.run() 2025-10-10T02:02:25.2263993Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:25.2264562Z (EngineCore_DP0 pid=12748) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:25.2265241Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:25.2265789Z (EngineCore_DP0 pid=12748) raise e 2025-10-10T02:02:25.2266378Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:25.2267028Z (EngineCore_DP0 pid=12748) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:25.2267472Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2268128Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:25.2268860Z (EngineCore_DP0 pid=12748) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:25.2269589Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:25.2270221Z (EngineCore_DP0 pid=12748) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:25.2270684Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2271387Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:25.2271982Z (EngineCore_DP0 pid=12748) self._init_executor() 2025-10-10T02:02:25.2272652Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:25.2273343Z (EngineCore_DP0 pid=12748) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:25.2274107Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:25.2274827Z (EngineCore_DP0 pid=12748) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:25.2275332Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2275986Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:25.2276581Z (EngineCore_DP0 pid=12748) return func(*args, **kwargs) 2025-10-10T02:02:25.2276986Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2277622Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:25.2278265Z (EngineCore_DP0 pid=12748) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:25.2278698Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2279451Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:25.2280116Z (EngineCore_DP0 pid=12748) module = importlib.import_module(module_name) 2025-10-10T02:02:25.2280562Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2281146Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:25.2281769Z (EngineCore_DP0 pid=12748) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:25.2282272Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2282776Z (EngineCore_DP0 pid=12748) File "", line 1387, in _gcd_import 2025-10-10T02:02:25.2283325Z (EngineCore_DP0 pid=12748) File "", line 1360, in _find_and_load 2025-10-10T02:02:25.2283914Z (EngineCore_DP0 pid=12748) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:25.2284493Z (EngineCore_DP0 pid=12748) File "", line 935, in _load_unlocked 2025-10-10T02:02:25.2285071Z (EngineCore_DP0 pid=12748) File "", line 999, in exec_module 2025-10-10T02:02:25.2285674Z (EngineCore_DP0 pid=12748) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:25.2286460Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:25.2287182Z (EngineCore_DP0 pid=12748) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:25.2287907Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:25.2288713Z (EngineCore_DP0 pid=12748) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:25.2289496Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:25.2290171Z (EngineCore_DP0 pid=12748) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:25.2290957Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:25.2291817Z (EngineCore_DP0 pid=12748) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:25.2292300Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2292992Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:25.2293682Z (EngineCore_DP0 pid=12748) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:25.2294117Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2294868Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:25.2295571Z (EngineCore_DP0 pid=12748) return _is_fa2_supported(device)[0] 2025-10-10T02:02:25.2295995Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2297032Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:25.2297753Z (EngineCore_DP0 pid=12748) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:25.2298219Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2298888Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:25.2299561Z (EngineCore_DP0 pid=12748) prop = get_device_properties(device) 2025-10-10T02:02:25.2299999Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:25.2300678Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:25.2301347Z (EngineCore_DP0 pid=12748) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:25.2301746Z (EngineCore_DP0 pid=12748) ^^^^^^^^^^^^ 2025-10-10T02:02:25.2302331Z (EngineCore_DP0 pid=12748) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:25.2302918Z (EngineCore_DP0 pid=12748) raise RuntimeError( 2025-10-10T02:02:25.2303607Z (EngineCore_DP0 pid=12748) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:25.6316708Z FAILED 2025-10-10T02:02:25.6445793Z models/test_initialization.py::test_can_initialize_large_subset[StableLMEpochForCausalLM] Fork a new process to run a test 12752 2025-10-10T02:02:25.6456512Z Fork a new process to run a test 0 2025-10-10T02:02:25.6725509Z INFO 10-10 02:02:25 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='StableLMEpochForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'stabilityai/stablelm-zephyr-3b'} 2025-10-10T02:02:25.8077962Z 2025-10-10T02:02:25.8079712Z config.json: 0% 0.00/599 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:26.7634095Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:26.7634654Z 2025-10-10T02:02:26.7635055Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:26.7635832Z 2025-10-10T02:02:26.7636636Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:26.7637230Z 2025-10-10T02:02:26.7637569Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:26.7637984Z 2025-10-10T02:02:26.7638309Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7638609Z 2025-10-10T02:02:26.7639399Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:26.7640000Z 2025-10-10T02:02:26.7640380Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:26.7640808Z 2025-10-10T02:02:26.7641392Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:26.7641859Z 2025-10-10T02:02:26.7642296Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:26.7642609Z 2025-10-10T02:02:26.7642916Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7643298Z 2025-10-10T02:02:26.7643875Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:26.7644405Z 2025-10-10T02:02:26.7644733Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] self._init_executor() 2025-10-10T02:02:26.7645056Z 2025-10-10T02:02:26.7645725Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:26.7646242Z 2025-10-10T02:02:26.7646703Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:26.7647041Z 2025-10-10T02:02:26.7647780Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:26.7648426Z 2025-10-10T02:02:26.7648834Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:26.7649285Z 2025-10-10T02:02:26.7649594Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7649892Z 2025-10-10T02:02:26.7650381Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:26.7650836Z 2025-10-10T02:02:26.7651222Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:26.7651509Z 2025-10-10T02:02:26.7651867Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7652134Z 2025-10-10T02:02:26.7652686Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:26.7653170Z 2025-10-10T02:02:26.7653468Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:26.7653805Z 2025-10-10T02:02:26.7654079Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7654348Z 2025-10-10T02:02:26.7654883Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:26.7655382Z 2025-10-10T02:02:26.7655687Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:26.7656035Z 2025-10-10T02:02:26.7656311Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7656599Z 2025-10-10T02:02:26.7657038Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:26.7657455Z 2025-10-10T02:02:26.7657798Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:26.7658125Z 2025-10-10T02:02:26.7658418Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7658696Z 2025-10-10T02:02:26.7659046Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:26.7659376Z 2025-10-10T02:02:26.7659732Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:26.7660070Z 2025-10-10T02:02:26.7660443Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:26.7660809Z 2025-10-10T02:02:26.7661150Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:26.7661486Z 2025-10-10T02:02:26.7661851Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:26.7662217Z 2025-10-10T02:02:26.7662599Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:26.7662970Z 2025-10-10T02:02:26.7663466Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:26.7663940Z 2025-10-10T02:02:26.7664302Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:26.7664652Z 2025-10-10T02:02:26.7665197Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:26.7665679Z 2025-10-10T02:02:26.7666112Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:26.7666526Z 2025-10-10T02:02:26.7667092Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:26.7667607Z 2025-10-10T02:02:26.7667903Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:26.7668266Z 2025-10-10T02:02:26.7668895Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:26.7669474Z 2025-10-10T02:02:26.7669843Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:26.7670201Z 2025-10-10T02:02:26.7670459Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7670725Z 2025-10-10T02:02:26.7671333Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:26.7671863Z 2025-10-10T02:02:26.7672169Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:26.7672469Z 2025-10-10T02:02:26.7672746Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7673022Z 2025-10-10T02:02:26.7673631Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:26.7674184Z 2025-10-10T02:02:26.7674463Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:26.7674755Z 2025-10-10T02:02:26.7675015Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7675288Z 2025-10-10T02:02:26.7675863Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:26.7676404Z 2025-10-10T02:02:26.7676725Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:26.7677045Z 2025-10-10T02:02:26.7677318Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7677595Z 2025-10-10T02:02:26.7678147Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:26.7678641Z 2025-10-10T02:02:26.7678924Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:26.7679351Z 2025-10-10T02:02:26.7679633Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7679915Z 2025-10-10T02:02:26.7680447Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:26.7680928Z 2025-10-10T02:02:26.7681310Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:26.7681660Z 2025-10-10T02:02:26.7681892Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:26.7682144Z 2025-10-10T02:02:26.7682628Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:26.7683087Z 2025-10-10T02:02:26.7683337Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] raise RuntimeError( 2025-10-10T02:02:26.7683643Z 2025-10-10T02:02:26.7684214Z (EngineCore_DP0 pid=12760) ERROR 10-10 02:02:26 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:26.7684926Z (EngineCore_DP0 pid=12760) Process EngineCore_DP0: 2025-10-10T02:02:26.7685334Z (EngineCore_DP0 pid=12760) Traceback (most recent call last): 2025-10-10T02:02:26.7685947Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:26.7686538Z (EngineCore_DP0 pid=12760) self.run() 2025-10-10T02:02:26.7687068Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:26.7687628Z (EngineCore_DP0 pid=12760) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:26.7688304Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:26.7688857Z (EngineCore_DP0 pid=12760) raise e 2025-10-10T02:02:26.7689444Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:26.7690102Z (EngineCore_DP0 pid=12760) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:26.7690556Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7691200Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:26.7691879Z (EngineCore_DP0 pid=12760) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:26.7692542Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:26.7693166Z (EngineCore_DP0 pid=12760) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:26.7693621Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7694274Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:26.7694860Z (EngineCore_DP0 pid=12760) self._init_executor() 2025-10-10T02:02:26.7695527Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:26.7696560Z (EngineCore_DP0 pid=12760) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:26.7697343Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:26.7698064Z (EngineCore_DP0 pid=12760) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:26.7698566Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7699292Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:26.7699957Z (EngineCore_DP0 pid=12760) return func(*args, **kwargs) 2025-10-10T02:02:26.7700349Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7701005Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:26.7701713Z (EngineCore_DP0 pid=12760) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:26.7702146Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7702814Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:26.7703480Z (EngineCore_DP0 pid=12760) module = importlib.import_module(module_name) 2025-10-10T02:02:26.7703934Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7704587Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:26.7705225Z (EngineCore_DP0 pid=12760) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:26.7705725Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7706228Z (EngineCore_DP0 pid=12760) File "", line 1387, in _gcd_import 2025-10-10T02:02:26.7706796Z (EngineCore_DP0 pid=12760) File "", line 1360, in _find_and_load 2025-10-10T02:02:26.7707389Z (EngineCore_DP0 pid=12760) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:26.7707977Z (EngineCore_DP0 pid=12760) File "", line 935, in _load_unlocked 2025-10-10T02:02:26.7708555Z (EngineCore_DP0 pid=12760) File "", line 999, in exec_module 2025-10-10T02:02:26.7709162Z (EngineCore_DP0 pid=12760) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:26.7709895Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:26.7710586Z (EngineCore_DP0 pid=12760) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:26.7711312Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:26.7712036Z (EngineCore_DP0 pid=12760) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:26.7712827Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:26.7713504Z (EngineCore_DP0 pid=12760) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:26.7714311Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:26.7715129Z (EngineCore_DP0 pid=12760) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:26.7715612Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7716303Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:26.7717043Z (EngineCore_DP0 pid=12760) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:26.7717528Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7718281Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:26.7718984Z (EngineCore_DP0 pid=12760) return _is_fa2_supported(device)[0] 2025-10-10T02:02:26.7719524Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7720263Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:26.7720985Z (EngineCore_DP0 pid=12760) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:26.7721449Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7722176Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:26.7722834Z (EngineCore_DP0 pid=12760) prop = get_device_properties(device) 2025-10-10T02:02:26.7723259Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:26.7723946Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:26.7724613Z (EngineCore_DP0 pid=12760) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:26.7725016Z (EngineCore_DP0 pid=12760) ^^^^^^^^^^^^ 2025-10-10T02:02:26.7725604Z (EngineCore_DP0 pid=12760) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:26.7726175Z (EngineCore_DP0 pid=12760) raise RuntimeError( 2025-10-10T02:02:26.7726870Z (EngineCore_DP0 pid=12760) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:27.1762247Z FAILED 2025-10-10T02:02:27.1891472Z models/test_initialization.py::test_can_initialize_large_subset[VoxtralForConditionalGeneration] Fork a new process to run a test 12764 2025-10-10T02:02:27.1904084Z Fork a new process to run a test 0 2025-10-10T02:02:27.1905746Z Model is not available online 2025-10-10T02:02:27.4933650Z PASSED 2025-10-10T02:02:27.5061301Z models/test_initialization.py::test_can_initialize_large_subset[Gemma3ForConditionalGeneration] Fork a new process to run a test 12765 2025-10-10T02:02:27.5073915Z Fork a new process to run a test 0 2025-10-10T02:02:27.5355998Z INFO 10-10 02:02:27 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Gemma3ForConditionalGeneration', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'google/gemma-3-4b-it'} 2025-10-10T02:02:27.6397918Z 2025-10-10T02:02:27.6400304Z config.json: 0% 0.00/855 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:38.6812841Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:38.6813531Z 2025-10-10T02:02:38.6814247Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:38.6814652Z 2025-10-10T02:02:38.6815482Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:38.6816088Z 2025-10-10T02:02:38.6816489Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:38.6816879Z 2025-10-10T02:02:38.6817366Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6817689Z 2025-10-10T02:02:38.6818252Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:38.6818946Z 2025-10-10T02:02:38.6819333Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:38.6819662Z 2025-10-10T02:02:38.6820258Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:38.6820739Z 2025-10-10T02:02:38.6821095Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:38.6821408Z 2025-10-10T02:02:38.6821687Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6822063Z 2025-10-10T02:02:38.6822784Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:38.6823419Z 2025-10-10T02:02:38.6823940Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] self._init_executor() 2025-10-10T02:02:38.6824402Z 2025-10-10T02:02:38.6825182Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:38.6825887Z 2025-10-10T02:02:38.6826238Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:38.6826580Z 2025-10-10T02:02:38.6827146Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:38.6827659Z 2025-10-10T02:02:38.6828016Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:38.6828351Z 2025-10-10T02:02:38.6828652Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6828935Z 2025-10-10T02:02:38.6829426Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:38.6829886Z 2025-10-10T02:02:38.6830154Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:38.6830431Z 2025-10-10T02:02:38.6830681Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6830944Z 2025-10-10T02:02:38.6831462Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:38.6831942Z 2025-10-10T02:02:38.6832302Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:38.6832653Z 2025-10-10T02:02:38.6832936Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6833210Z 2025-10-10T02:02:38.6833762Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:38.6834296Z 2025-10-10T02:02:38.6834610Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:38.6834910Z 2025-10-10T02:02:38.6835181Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6835458Z 2025-10-10T02:02:38.6835900Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:38.6836322Z 2025-10-10T02:02:38.6836717Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:38.6837065Z 2025-10-10T02:02:38.6837354Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6837641Z 2025-10-10T02:02:38.6837982Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:38.6838309Z 2025-10-10T02:02:38.6838663Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:38.6838999Z 2025-10-10T02:02:38.6839536Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:38.6839895Z 2025-10-10T02:02:38.6840246Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:38.6840580Z 2025-10-10T02:02:38.6840937Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:38.6841291Z 2025-10-10T02:02:38.6841665Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:38.6842029Z 2025-10-10T02:02:38.6842528Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:38.6843001Z 2025-10-10T02:02:38.6843363Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:38.6843704Z 2025-10-10T02:02:38.6844244Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:38.6844725Z 2025-10-10T02:02:38.6845105Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:38.6845466Z 2025-10-10T02:02:38.6846017Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:38.6846524Z 2025-10-10T02:02:38.6846876Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:38.6847207Z 2025-10-10T02:02:38.6847853Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:38.6848434Z 2025-10-10T02:02:38.6848801Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:38.6849199Z 2025-10-10T02:02:38.6849455Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6849723Z 2025-10-10T02:02:38.6850287Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:38.6850821Z 2025-10-10T02:02:38.6851124Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:38.6851424Z 2025-10-10T02:02:38.6851734Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6852010Z 2025-10-10T02:02:38.6852626Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:38.6853189Z 2025-10-10T02:02:38.6853477Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:38.6853761Z 2025-10-10T02:02:38.6854027Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6854292Z 2025-10-10T02:02:38.6854867Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:38.6855400Z 2025-10-10T02:02:38.6855719Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:38.6856036Z 2025-10-10T02:02:38.6856308Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6856605Z 2025-10-10T02:02:38.6864312Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:38.6864854Z 2025-10-10T02:02:38.6865224Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:38.6865524Z 2025-10-10T02:02:38.6865820Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6866097Z 2025-10-10T02:02:38.6866641Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:38.6867153Z 2025-10-10T02:02:38.6867470Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:38.6867795Z 2025-10-10T02:02:38.6868025Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:38.6868279Z 2025-10-10T02:02:38.6868783Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:38.6869248Z 2025-10-10T02:02:38.6869573Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] raise RuntimeError( 2025-10-10T02:02:38.6869876Z 2025-10-10T02:02:38.6870461Z (EngineCore_DP0 pid=12844) ERROR 10-10 02:02:38 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:38.6871173Z (EngineCore_DP0 pid=12844) Process EngineCore_DP0: 2025-10-10T02:02:38.6871571Z (EngineCore_DP0 pid=12844) Traceback (most recent call last): 2025-10-10T02:02:38.6872270Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:38.6872790Z (EngineCore_DP0 pid=12844) self.run() 2025-10-10T02:02:38.6873322Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:38.6873903Z (EngineCore_DP0 pid=12844) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:38.6874622Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:38.6875183Z (EngineCore_DP0 pid=12844) raise e 2025-10-10T02:02:38.6875782Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:38.6876429Z (EngineCore_DP0 pid=12844) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:38.6876878Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6877506Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:38.6878154Z (EngineCore_DP0 pid=12844) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:38.6878816Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:38.6879525Z (EngineCore_DP0 pid=12844) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:38.6879982Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6880638Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:38.6881231Z (EngineCore_DP0 pid=12844) self._init_executor() 2025-10-10T02:02:38.6881916Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:38.6882606Z (EngineCore_DP0 pid=12844) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:38.6883337Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:38.6884056Z (EngineCore_DP0 pid=12844) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:38.6884558Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6885198Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:38.6885793Z (EngineCore_DP0 pid=12844) return func(*args, **kwargs) 2025-10-10T02:02:38.6886177Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6886864Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:38.6887546Z (EngineCore_DP0 pid=12844) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:38.6887975Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6888649Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:38.6889314Z (EngineCore_DP0 pid=12844) module = importlib.import_module(module_name) 2025-10-10T02:02:38.6889800Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6890392Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:38.6891032Z (EngineCore_DP0 pid=12844) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:38.6891522Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6892060Z (EngineCore_DP0 pid=12844) File "", line 1387, in _gcd_import 2025-10-10T02:02:38.6892623Z (EngineCore_DP0 pid=12844) File "", line 1360, in _find_and_load 2025-10-10T02:02:38.6893219Z (EngineCore_DP0 pid=12844) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:38.6893807Z (EngineCore_DP0 pid=12844) File "", line 935, in _load_unlocked 2025-10-10T02:02:38.6894373Z (EngineCore_DP0 pid=12844) File "", line 999, in exec_module 2025-10-10T02:02:38.6894980Z (EngineCore_DP0 pid=12844) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:38.6895715Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:38.6896635Z (EngineCore_DP0 pid=12844) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:38.6897370Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:38.6898092Z (EngineCore_DP0 pid=12844) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:38.6898885Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:38.6899562Z (EngineCore_DP0 pid=12844) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:38.6900362Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:38.6901184Z (EngineCore_DP0 pid=12844) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:38.6901669Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6902356Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:38.6903042Z (EngineCore_DP0 pid=12844) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:38.6903480Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6904248Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:38.6905053Z (EngineCore_DP0 pid=12844) return _is_fa2_supported(device)[0] 2025-10-10T02:02:38.6905542Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6906264Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:38.6907000Z (EngineCore_DP0 pid=12844) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:38.6907524Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6908198Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:38.6908833Z (EngineCore_DP0 pid=12844) prop = get_device_properties(device) 2025-10-10T02:02:38.6909253Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:38.6909910Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:38.6910653Z (EngineCore_DP0 pid=12844) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:38.6911068Z (EngineCore_DP0 pid=12844) ^^^^^^^^^^^^ 2025-10-10T02:02:38.6911647Z (EngineCore_DP0 pid=12844) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:38.6912226Z (EngineCore_DP0 pid=12844) raise RuntimeError( 2025-10-10T02:02:38.6912919Z (EngineCore_DP0 pid=12844) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:39.1345004Z FAILED 2025-10-10T02:02:39.1473884Z models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeSharedForCausalLM] Fork a new process to run a test 12848 2025-10-10T02:02:39.1486153Z Fork a new process to run a test 0 2025-10-10T02:02:39.1755712Z INFO 10-10 02:02:39 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GraniteMoeSharedForCausalLM', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'ibm-research/moe-7b-1b-active-shared-experts'} 2025-10-10T02:02:39.3056415Z 2025-10-10T02:02:39.3059162Z config.json: 0% 0.00/901 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:47.2199021Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:47.2199554Z 2025-10-10T02:02:47.2199951Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:47.2200329Z 2025-10-10T02:02:47.2201037Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:47.2201659Z 2025-10-10T02:02:47.2202053Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:47.2202431Z 2025-10-10T02:02:47.2202777Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2203149Z 2025-10-10T02:02:47.2203764Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:47.2204328Z 2025-10-10T02:02:47.2204756Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:47.2205384Z 2025-10-10T02:02:47.2205994Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:47.2206454Z 2025-10-10T02:02:47.2206770Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:47.2207075Z 2025-10-10T02:02:47.2207359Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2207757Z 2025-10-10T02:02:47.2208289Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:47.2208761Z 2025-10-10T02:02:47.2209070Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] self._init_executor() 2025-10-10T02:02:47.2209542Z 2025-10-10T02:02:47.2210540Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:47.2211221Z 2025-10-10T02:02:47.2211720Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:47.2212094Z 2025-10-10T02:02:47.2212880Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:47.2213412Z 2025-10-10T02:02:47.2213786Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:47.2214125Z 2025-10-10T02:02:47.2214465Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2214813Z 2025-10-10T02:02:47.2215399Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:47.2215948Z 2025-10-10T02:02:47.2216282Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:47.2216626Z 2025-10-10T02:02:47.2216901Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2217164Z 2025-10-10T02:02:47.2217702Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:47.2218184Z 2025-10-10T02:02:47.2218476Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:47.2218778Z 2025-10-10T02:02:47.2219048Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2219326Z 2025-10-10T02:02:47.2219889Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:47.2220376Z 2025-10-10T02:02:47.2220684Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:47.2220985Z 2025-10-10T02:02:47.2221257Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2221537Z 2025-10-10T02:02:47.2222054Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:47.2222518Z 2025-10-10T02:02:47.2222878Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:47.2223218Z 2025-10-10T02:02:47.2223507Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2223835Z 2025-10-10T02:02:47.2224192Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:47.2224568Z 2025-10-10T02:02:47.2224989Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:47.2225407Z 2025-10-10T02:02:47.2225885Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:47.2226330Z 2025-10-10T02:02:47.2226726Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:47.2227066Z 2025-10-10T02:02:47.2227430Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:47.2227791Z 2025-10-10T02:02:47.2228167Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:47.2228535Z 2025-10-10T02:02:47.2229048Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:47.2229524Z 2025-10-10T02:02:47.2229880Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:47.2230235Z 2025-10-10T02:02:47.2230763Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:47.2231243Z 2025-10-10T02:02:47.2231622Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:47.2231987Z 2025-10-10T02:02:47.2232540Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:47.2233051Z 2025-10-10T02:02:47.2233351Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:47.2233649Z 2025-10-10T02:02:47.2234280Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:47.2234882Z 2025-10-10T02:02:47.2235248Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:47.2235605Z 2025-10-10T02:02:47.2235879Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2236148Z 2025-10-10T02:02:47.2236716Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:47.2237238Z 2025-10-10T02:02:47.2237585Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:47.2237930Z 2025-10-10T02:02:47.2238217Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2238488Z 2025-10-10T02:02:47.2239191Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:47.2239799Z 2025-10-10T02:02:47.2240095Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:47.2240383Z 2025-10-10T02:02:47.2240653Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2240918Z 2025-10-10T02:02:47.2241492Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:47.2242028Z 2025-10-10T02:02:47.2242398Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:47.2242721Z 2025-10-10T02:02:47.2242999Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2243276Z 2025-10-10T02:02:47.2243800Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:47.2244291Z 2025-10-10T02:02:47.2244574Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:47.2244860Z 2025-10-10T02:02:47.2245141Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2245428Z 2025-10-10T02:02:47.2245960Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:47.2246445Z 2025-10-10T02:02:47.2246760Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:47.2247065Z 2025-10-10T02:02:47.2247289Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:47.2247538Z 2025-10-10T02:02:47.2248034Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:47.2248505Z 2025-10-10T02:02:47.2248757Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] raise RuntimeError( 2025-10-10T02:02:47.2249024Z 2025-10-10T02:02:47.2249590Z (EngineCore_DP0 pid=12912) ERROR 10-10 02:02:47 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:47.2250289Z (EngineCore_DP0 pid=12912) Process EngineCore_DP0: 2025-10-10T02:02:47.2250688Z (EngineCore_DP0 pid=12912) Traceback (most recent call last): 2025-10-10T02:02:47.2251308Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:47.2251827Z (EngineCore_DP0 pid=12912) self.run() 2025-10-10T02:02:47.2252353Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:47.2252919Z (EngineCore_DP0 pid=12912) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:47.2253672Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:47.2254293Z (EngineCore_DP0 pid=12912) raise e 2025-10-10T02:02:47.2254902Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:47.2255537Z (EngineCore_DP0 pid=12912) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:47.2256044Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2256674Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:47.2257323Z (EngineCore_DP0 pid=12912) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:47.2257994Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:47.2258673Z (EngineCore_DP0 pid=12912) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:47.2259135Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2259789Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:47.2260404Z (EngineCore_DP0 pid=12912) self._init_executor() 2025-10-10T02:02:47.2261079Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:47.2261880Z (EngineCore_DP0 pid=12912) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:47.2262612Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:47.2263348Z (EngineCore_DP0 pid=12912) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:47.2263849Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2264486Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:47.2265087Z (EngineCore_DP0 pid=12912) return func(*args, **kwargs) 2025-10-10T02:02:47.2265471Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2266110Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:47.2266754Z (EngineCore_DP0 pid=12912) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:47.2267191Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2267876Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:47.2268540Z (EngineCore_DP0 pid=12912) module = importlib.import_module(module_name) 2025-10-10T02:02:47.2268990Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2269577Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:47.2270202Z (EngineCore_DP0 pid=12912) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:47.2270759Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2271309Z (EngineCore_DP0 pid=12912) File "", line 1387, in _gcd_import 2025-10-10T02:02:47.2271870Z (EngineCore_DP0 pid=12912) File "", line 1360, in _find_and_load 2025-10-10T02:02:47.2272459Z (EngineCore_DP0 pid=12912) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:47.2273043Z (EngineCore_DP0 pid=12912) File "", line 935, in _load_unlocked 2025-10-10T02:02:47.2273672Z (EngineCore_DP0 pid=12912) File "", line 999, in exec_module 2025-10-10T02:02:47.2274281Z (EngineCore_DP0 pid=12912) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:47.2275023Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:47.2275706Z (EngineCore_DP0 pid=12912) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:47.2276476Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:47.2277205Z (EngineCore_DP0 pid=12912) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:47.2278001Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:47.2278674Z (EngineCore_DP0 pid=12912) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:47.2279541Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:47.2280378Z (EngineCore_DP0 pid=12912) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:47.2280881Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2281575Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:47.2282263Z (EngineCore_DP0 pid=12912) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:47.2282709Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2283462Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:47.2284168Z (EngineCore_DP0 pid=12912) return _is_fa2_supported(device)[0] 2025-10-10T02:02:47.2284588Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2285313Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:47.2286035Z (EngineCore_DP0 pid=12912) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:47.2286505Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2287182Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:47.2287820Z (EngineCore_DP0 pid=12912) prop = get_device_properties(device) 2025-10-10T02:02:47.2288240Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:47.2288962Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:47.2289694Z (EngineCore_DP0 pid=12912) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:47.2290099Z (EngineCore_DP0 pid=12912) ^^^^^^^^^^^^ 2025-10-10T02:02:47.2290676Z (EngineCore_DP0 pid=12912) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:47.2291291Z (EngineCore_DP0 pid=12912) raise RuntimeError( 2025-10-10T02:02:47.2291983Z (EngineCore_DP0 pid=12912) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:47.6442728Z FAILED 2025-10-10T02:02:47.6570942Z models/test_initialization.py::test_implicit_converted_models[GemmaForSequenceClassification] Fork a new process to run a test 12916 2025-10-10T02:02:47.6582420Z Fork a new process to run a test 0 2025-10-10T02:02:47.6854167Z INFO 10-10 02:02:47 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='GemmaForSequenceClassification', exist_overrides={'architectures': ['GemmaForSequenceClassification'], 'classifier_from_token': ['Yes'], 'method': 'no_post_processing'}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'BAAI/bge-reranker-v2-gemma'} 2025-10-10T02:02:47.7654950Z 2025-10-10T02:02:47.7656399Z config.json: 0% 0.00/659 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:49.9541820Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:49.9542403Z 2025-10-10T02:02:49.9542996Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:49.9543453Z 2025-10-10T02:02:49.9544161Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:49.9544815Z 2025-10-10T02:02:49.9545232Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:49.9545649Z 2025-10-10T02:02:49.9546031Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9546376Z 2025-10-10T02:02:49.9546996Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:49.9547554Z 2025-10-10T02:02:49.9547936Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:49.9548268Z 2025-10-10T02:02:49.9548861Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:49.9549311Z 2025-10-10T02:02:49.9549846Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:49.9550287Z 2025-10-10T02:02:49.9550596Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9550883Z 2025-10-10T02:02:49.9551394Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:49.9551960Z 2025-10-10T02:02:49.9552224Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] self._init_executor() 2025-10-10T02:02:49.9552606Z 2025-10-10T02:02:49.9553292Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:49.9554106Z 2025-10-10T02:02:49.9554601Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:49.9555200Z 2025-10-10T02:02:49.9555957Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:49.9556500Z 2025-10-10T02:02:49.9556864Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:49.9557212Z 2025-10-10T02:02:49.9557511Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9557795Z 2025-10-10T02:02:49.9558298Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:49.9558757Z 2025-10-10T02:02:49.9559023Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:49.9559428Z 2025-10-10T02:02:49.9559700Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9559959Z 2025-10-10T02:02:49.9560480Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:49.9560952Z 2025-10-10T02:02:49.9561244Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:49.9561534Z 2025-10-10T02:02:49.9561801Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9562078Z 2025-10-10T02:02:49.9562609Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:49.9563100Z 2025-10-10T02:02:49.9563402Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:49.9563705Z 2025-10-10T02:02:49.9563978Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9564261Z 2025-10-10T02:02:49.9564708Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:49.9565136Z 2025-10-10T02:02:49.9565495Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:49.9565903Z 2025-10-10T02:02:49.9566317Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9566621Z 2025-10-10T02:02:49.9566975Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:49.9567305Z 2025-10-10T02:02:49.9567654Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:49.9568052Z 2025-10-10T02:02:49.9568431Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:49.9568796Z 2025-10-10T02:02:49.9569140Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:49.9569482Z 2025-10-10T02:02:49.9569850Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:49.9570247Z 2025-10-10T02:02:49.9570637Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:49.9570999Z 2025-10-10T02:02:49.9571512Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:49.9571982Z 2025-10-10T02:02:49.9572337Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:49.9572675Z 2025-10-10T02:02:49.9573207Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:49.9573695Z 2025-10-10T02:02:49.9574072Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:49.9574457Z 2025-10-10T02:02:49.9575020Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:49.9575545Z 2025-10-10T02:02:49.9575841Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:49.9576141Z 2025-10-10T02:02:49.9576781Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:49.9577369Z 2025-10-10T02:02:49.9577736Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:49.9578094Z 2025-10-10T02:02:49.9578350Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9578612Z 2025-10-10T02:02:49.9579177Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:49.9579695Z 2025-10-10T02:02:49.9579997Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:49.9580292Z 2025-10-10T02:02:49.9580569Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9580849Z 2025-10-10T02:02:49.9581513Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:49.9582106Z 2025-10-10T02:02:49.9582404Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:49.9582694Z 2025-10-10T02:02:49.9582954Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9583271Z 2025-10-10T02:02:49.9583849Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:49.9584383Z 2025-10-10T02:02:49.9584706Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:49.9585017Z 2025-10-10T02:02:49.9585312Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9585624Z 2025-10-10T02:02:49.9586160Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:49.9586644Z 2025-10-10T02:02:49.9586935Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:49.9587222Z 2025-10-10T02:02:49.9587489Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9587777Z 2025-10-10T02:02:49.9588305Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:49.9588803Z 2025-10-10T02:02:49.9589117Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:49.9589426Z 2025-10-10T02:02:49.9589655Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:49.9589905Z 2025-10-10T02:02:49.9590385Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:49.9590833Z 2025-10-10T02:02:49.9591085Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] raise RuntimeError( 2025-10-10T02:02:49.9591340Z 2025-10-10T02:02:49.9591913Z (EngineCore_DP0 pid=12945) ERROR 10-10 02:02:49 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:49.9592611Z (EngineCore_DP0 pid=12945) Process EngineCore_DP0: 2025-10-10T02:02:49.9593040Z (EngineCore_DP0 pid=12945) Traceback (most recent call last): 2025-10-10T02:02:49.9593647Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:49.9594174Z (EngineCore_DP0 pid=12945) self.run() 2025-10-10T02:02:49.9594715Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:49.9595315Z (EngineCore_DP0 pid=12945) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:49.9595992Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:49.9597137Z (EngineCore_DP0 pid=12945) raise e 2025-10-10T02:02:49.9597923Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:49.9598694Z (EngineCore_DP0 pid=12945) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:49.9599231Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9599867Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:49.9600604Z (EngineCore_DP0 pid=12945) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:49.9601281Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:49.9601929Z (EngineCore_DP0 pid=12945) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:49.9602387Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9603128Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:49.9603723Z (EngineCore_DP0 pid=12945) self._init_executor() 2025-10-10T02:02:49.9604402Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:49.9605098Z (EngineCore_DP0 pid=12945) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:49.9605822Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:49.9606538Z (EngineCore_DP0 pid=12945) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:49.9607036Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9607688Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:49.9608276Z (EngineCore_DP0 pid=12945) return func(*args, **kwargs) 2025-10-10T02:02:49.9608664Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9609301Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:49.9609932Z (EngineCore_DP0 pid=12945) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:49.9610357Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9611025Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:49.9611685Z (EngineCore_DP0 pid=12945) module = importlib.import_module(module_name) 2025-10-10T02:02:49.9612126Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9612710Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:49.9613342Z (EngineCore_DP0 pid=12945) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:49.9613835Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9614327Z (EngineCore_DP0 pid=12945) File "", line 1387, in _gcd_import 2025-10-10T02:02:49.9614937Z (EngineCore_DP0 pid=12945) File "", line 1360, in _find_and_load 2025-10-10T02:02:49.9615564Z (EngineCore_DP0 pid=12945) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:49.9616146Z (EngineCore_DP0 pid=12945) File "", line 935, in _load_unlocked 2025-10-10T02:02:49.9616720Z (EngineCore_DP0 pid=12945) File "", line 999, in exec_module 2025-10-10T02:02:49.9617323Z (EngineCore_DP0 pid=12945) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:49.9618090Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:49.9618770Z (EngineCore_DP0 pid=12945) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:49.9619499Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:49.9620266Z (EngineCore_DP0 pid=12945) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:49.9621051Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:49.9621739Z (EngineCore_DP0 pid=12945) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:49.9622529Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:49.9623344Z (EngineCore_DP0 pid=12945) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:49.9623826Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9624525Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:49.9625236Z (EngineCore_DP0 pid=12945) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:49.9625684Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9626436Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:49.9627183Z (EngineCore_DP0 pid=12945) return _is_fa2_supported(device)[0] 2025-10-10T02:02:49.9627615Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9628348Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:49.9629065Z (EngineCore_DP0 pid=12945) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:49.9629532Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9630204Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:49.9630843Z (EngineCore_DP0 pid=12945) prop = get_device_properties(device) 2025-10-10T02:02:49.9631273Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:49.9631934Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:49.9632598Z (EngineCore_DP0 pid=12945) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:49.9633038Z (EngineCore_DP0 pid=12945) ^^^^^^^^^^^^ 2025-10-10T02:02:49.9633666Z (EngineCore_DP0 pid=12945) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:49.9634229Z (EngineCore_DP0 pid=12945) raise RuntimeError( 2025-10-10T02:02:49.9634917Z (EngineCore_DP0 pid=12945) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:50.3773358Z FAILED 2025-10-10T02:02:50.3902595Z models/test_initialization.py::test_implicit_converted_models[LlamaForSequenceClassification] Fork a new process to run a test 12949 2025-10-10T02:02:50.3914293Z Fork a new process to run a test 0 2025-10-10T02:02:50.4187067Z INFO 10-10 02:02:50 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='LlamaForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'Skywork/Skywork-Reward-V2-Llama-3.2-1B'} 2025-10-10T02:02:50.4981521Z 2025-10-10T02:02:50.4982620Z config.json: 0% 0.00/991 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:52.0265141Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:52.0265725Z 2025-10-10T02:02:52.0266347Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:52.0266831Z 2025-10-10T02:02:52.0267576Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:52.0268212Z 2025-10-10T02:02:52.0268617Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:52.0268994Z 2025-10-10T02:02:52.0269351Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0269664Z 2025-10-10T02:02:52.0270150Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:52.0270599Z 2025-10-10T02:02:52.0271157Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:52.0271543Z 2025-10-10T02:02:52.0272238Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:52.0272693Z 2025-10-10T02:02:52.0273053Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:52.0273364Z 2025-10-10T02:02:52.0273661Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0273942Z 2025-10-10T02:02:52.0274471Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:52.0275334Z 2025-10-10T02:02:52.0275840Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] self._init_executor() 2025-10-10T02:02:52.0276489Z 2025-10-10T02:02:52.0277472Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:52.0278233Z 2025-10-10T02:02:52.0278737Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:52.0279503Z 2025-10-10T02:02:52.0280498Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:52.0281039Z 2025-10-10T02:02:52.0281423Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:52.0281905Z 2025-10-10T02:02:52.0282410Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0282899Z 2025-10-10T02:02:52.0283908Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:52.0284684Z 2025-10-10T02:02:52.0285121Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:52.0285548Z 2025-10-10T02:02:52.0285944Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0286355Z 2025-10-10T02:02:52.0287165Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:52.0287915Z 2025-10-10T02:02:52.0288366Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:52.0288825Z 2025-10-10T02:02:52.0289259Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0289683Z 2025-10-10T02:02:52.0290512Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:52.0291288Z 2025-10-10T02:02:52.0291774Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:52.0292252Z 2025-10-10T02:02:52.0292712Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0293175Z 2025-10-10T02:02:52.0293920Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:52.0294628Z 2025-10-10T02:02:52.0295207Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:52.0295794Z 2025-10-10T02:02:52.0296520Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0297036Z 2025-10-10T02:02:52.0297621Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:52.0298178Z 2025-10-10T02:02:52.0298766Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:52.0299349Z 2025-10-10T02:02:52.0300098Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:52.0300784Z 2025-10-10T02:02:52.0301379Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:52.0301945Z 2025-10-10T02:02:52.0302574Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:52.0303175Z 2025-10-10T02:02:52.0303931Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:52.0304567Z 2025-10-10T02:02:52.0305449Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:52.0306265Z 2025-10-10T02:02:52.0306880Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:52.0307489Z 2025-10-10T02:02:52.0308592Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:52.0309465Z 2025-10-10T02:02:52.0310125Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:52.0310761Z 2025-10-10T02:02:52.0311699Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:52.0312555Z 2025-10-10T02:02:52.0313062Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:52.0313564Z 2025-10-10T02:02:52.0314650Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:52.0315628Z 2025-10-10T02:02:52.0316250Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:52.0316842Z 2025-10-10T02:02:52.0317272Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0317725Z 2025-10-10T02:02:52.0318697Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:52.0319758Z 2025-10-10T02:02:52.0320303Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:52.0320835Z 2025-10-10T02:02:52.0321296Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0321795Z 2025-10-10T02:02:52.0322897Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:52.0323865Z 2025-10-10T02:02:52.0324341Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:52.0324804Z 2025-10-10T02:02:52.0325234Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0325669Z 2025-10-10T02:02:52.0326719Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:52.0327673Z 2025-10-10T02:02:52.0328226Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:52.0328768Z 2025-10-10T02:02:52.0329232Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0329818Z 2025-10-10T02:02:52.0330704Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:52.0331521Z 2025-10-10T02:02:52.0332015Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:52.0332510Z 2025-10-10T02:02:52.0332993Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0333473Z 2025-10-10T02:02:52.0334499Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:52.0335336Z 2025-10-10T02:02:52.0335873Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:52.0336381Z 2025-10-10T02:02:52.0336768Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:52.0337174Z 2025-10-10T02:02:52.0337993Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:52.0338766Z 2025-10-10T02:02:52.0339182Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] raise RuntimeError( 2025-10-10T02:02:52.0339624Z 2025-10-10T02:02:52.0340582Z (EngineCore_DP0 pid=12977) ERROR 10-10 02:02:52 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:52.0341743Z (EngineCore_DP0 pid=12977) Process EngineCore_DP0: 2025-10-10T02:02:52.0342434Z (EngineCore_DP0 pid=12977) Traceback (most recent call last): 2025-10-10T02:02:52.0343443Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:52.0344355Z (EngineCore_DP0 pid=12977) self.run() 2025-10-10T02:02:52.0345312Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:52.0346335Z (EngineCore_DP0 pid=12977) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:52.0347512Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:52.0348440Z (EngineCore_DP0 pid=12977) raise e 2025-10-10T02:02:52.0349464Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:52.0350544Z (EngineCore_DP0 pid=12977) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:52.0351286Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0352330Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:52.0353431Z (EngineCore_DP0 pid=12977) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:52.0354621Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:52.0355769Z (EngineCore_DP0 pid=12977) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:52.0356519Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0357740Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:52.0358882Z (EngineCore_DP0 pid=12977) self._init_executor() 2025-10-10T02:02:52.0360018Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:52.0361131Z (EngineCore_DP0 pid=12977) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:52.0362411Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:52.0363735Z (EngineCore_DP0 pid=12977) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:52.0364594Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0365690Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:52.0366699Z (EngineCore_DP0 pid=12977) return func(*args, **kwargs) 2025-10-10T02:02:52.0367350Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0368441Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:52.0369583Z (EngineCore_DP0 pid=12977) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:52.0370265Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0371478Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:52.0372669Z (EngineCore_DP0 pid=12977) module = importlib.import_module(module_name) 2025-10-10T02:02:52.0373408Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0374427Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:52.0375517Z (EngineCore_DP0 pid=12977) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:52.0376336Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0377108Z (EngineCore_DP0 pid=12977) File "", line 1387, in _gcd_import 2025-10-10T02:02:52.0378072Z (EngineCore_DP0 pid=12977) File "", line 1360, in _find_and_load 2025-10-10T02:02:52.0379120Z (EngineCore_DP0 pid=12977) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:52.0380154Z (EngineCore_DP0 pid=12977) File "", line 935, in _load_unlocked 2025-10-10T02:02:52.0381235Z (EngineCore_DP0 pid=12977) File "", line 999, in exec_module 2025-10-10T02:02:52.0382351Z (EngineCore_DP0 pid=12977) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:52.0383717Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:52.0384953Z (EngineCore_DP0 pid=12977) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:52.0386093Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:52.0387270Z (EngineCore_DP0 pid=12977) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:52.0388722Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:52.0389935Z (EngineCore_DP0 pid=12977) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:52.0391400Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:52.0392795Z (EngineCore_DP0 pid=12977) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:52.0393650Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0394839Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:52.0396003Z (EngineCore_DP0 pid=12977) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:52.0396912Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0398227Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:52.0399519Z (EngineCore_DP0 pid=12977) return _is_fa2_supported(device)[0] 2025-10-10T02:02:52.0400209Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0401500Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:52.0402721Z (EngineCore_DP0 pid=12977) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:52.0403542Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0404754Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:52.0405907Z (EngineCore_DP0 pid=12977) prop = get_device_properties(device) 2025-10-10T02:02:52.0406655Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:52.0407861Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:52.0409077Z (EngineCore_DP0 pid=12977) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:52.0409787Z (EngineCore_DP0 pid=12977) ^^^^^^^^^^^^ 2025-10-10T02:02:52.0410832Z (EngineCore_DP0 pid=12977) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:52.0411863Z (EngineCore_DP0 pid=12977) raise RuntimeError( 2025-10-10T02:02:52.0413123Z (EngineCore_DP0 pid=12977) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:52.4565103Z FAILED 2025-10-10T02:02:52.4693330Z models/test_initialization.py::test_implicit_converted_models[Qwen2ForSequenceClassification] Fork a new process to run a test 12981 2025-10-10T02:02:52.4707148Z Fork a new process to run a test 0 2025-10-10T02:02:52.4985109Z INFO 10-10 02:02:52 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen2ForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'jason9693/Qwen2.5-1.5B-apeach'} 2025-10-10T02:02:52.6448229Z 2025-10-10T02:02:52.6450018Z config.json: 0% 0.00/950 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:54.3374439Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:54.3375062Z 2025-10-10T02:02:54.3375514Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:54.3375891Z 2025-10-10T02:02:54.3376579Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:54.3377176Z 2025-10-10T02:02:54.3377579Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:54.3377960Z 2025-10-10T02:02:54.3378332Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3378681Z 2025-10-10T02:02:54.3379284Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:54.3379832Z 2025-10-10T02:02:54.3380315Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:54.3380918Z 2025-10-10T02:02:54.3381579Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:54.3382041Z 2025-10-10T02:02:54.3382369Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:54.3382687Z 2025-10-10T02:02:54.3382979Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3383267Z 2025-10-10T02:02:54.3383771Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:54.3384616Z 2025-10-10T02:02:54.3385120Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] self._init_executor() 2025-10-10T02:02:54.3385607Z 2025-10-10T02:02:54.3386624Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:54.3387519Z 2025-10-10T02:02:54.3388158Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:54.3388594Z 2025-10-10T02:02:54.3389287Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:54.3389800Z 2025-10-10T02:02:54.3390164Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:54.3390624Z 2025-10-10T02:02:54.3390932Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3391221Z 2025-10-10T02:02:54.3391717Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:54.3392176Z 2025-10-10T02:02:54.3392453Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:54.3392728Z 2025-10-10T02:02:54.3393114Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3393381Z 2025-10-10T02:02:54.3393908Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:54.3394389Z 2025-10-10T02:02:54.3394687Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:54.3394975Z 2025-10-10T02:02:54.3395242Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3395521Z 2025-10-10T02:02:54.3396045Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:54.3396931Z 2025-10-10T02:02:54.3397276Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:54.3397597Z 2025-10-10T02:02:54.3397883Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3398168Z 2025-10-10T02:02:54.3398617Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:54.3399166Z 2025-10-10T02:02:54.3399533Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:54.3399864Z 2025-10-10T02:02:54.3400171Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3400457Z 2025-10-10T02:02:54.3400822Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:54.3401152Z 2025-10-10T02:02:54.3401512Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:54.3401864Z 2025-10-10T02:02:54.3402251Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:54.3402616Z 2025-10-10T02:02:54.3402972Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:54.3403311Z 2025-10-10T02:02:54.3403777Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:54.3404217Z 2025-10-10T02:02:54.3404615Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:54.3404978Z 2025-10-10T02:02:54.3405486Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:54.3406074Z 2025-10-10T02:02:54.3406451Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:54.3406785Z 2025-10-10T02:02:54.3407306Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:54.3407795Z 2025-10-10T02:02:54.3408187Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:54.3408615Z 2025-10-10T02:02:54.3409194Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:54.3409709Z 2025-10-10T02:02:54.3410020Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:54.3410327Z 2025-10-10T02:02:54.3410963Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:54.3411546Z 2025-10-10T02:02:54.3411928Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:54.3412288Z 2025-10-10T02:02:54.3412544Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3412807Z 2025-10-10T02:02:54.3413387Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:54.3413907Z 2025-10-10T02:02:54.3414211Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:54.3414506Z 2025-10-10T02:02:54.3414794Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3415070Z 2025-10-10T02:02:54.3415672Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:54.3416229Z 2025-10-10T02:02:54.3416525Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:54.3416815Z 2025-10-10T02:02:54.3417079Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3417355Z 2025-10-10T02:02:54.3417932Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:54.3418459Z 2025-10-10T02:02:54.3418782Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:54.3419094Z 2025-10-10T02:02:54.3419430Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3419740Z 2025-10-10T02:02:54.3420294Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:54.3420778Z 2025-10-10T02:02:54.3421079Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:54.3421419Z 2025-10-10T02:02:54.3421702Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3421985Z 2025-10-10T02:02:54.3422519Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:54.3423004Z 2025-10-10T02:02:54.3423321Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:54.3423634Z 2025-10-10T02:02:54.3423908Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:54.3424172Z 2025-10-10T02:02:54.3424655Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:54.3425109Z 2025-10-10T02:02:54.3425365Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] raise RuntimeError( 2025-10-10T02:02:54.3425627Z 2025-10-10T02:02:54.3426205Z (EngineCore_DP0 pid=13011) ERROR 10-10 02:02:54 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:54.3426928Z (EngineCore_DP0 pid=13011) Process EngineCore_DP0: 2025-10-10T02:02:54.3427342Z (EngineCore_DP0 pid=13011) Traceback (most recent call last): 2025-10-10T02:02:54.3427956Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:54.3428487Z (EngineCore_DP0 pid=13011) self.run() 2025-10-10T02:02:54.3429031Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:54.3429604Z (EngineCore_DP0 pid=13011) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:54.3430272Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:54.3430814Z (EngineCore_DP0 pid=13011) raise e 2025-10-10T02:02:54.3431402Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:54.3432040Z (EngineCore_DP0 pid=13011) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:54.3432504Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3433139Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:54.3433779Z (EngineCore_DP0 pid=13011) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:54.3434437Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:54.3435073Z (EngineCore_DP0 pid=13011) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:54.3435579Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3436301Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:54.3436885Z (EngineCore_DP0 pid=13011) self._init_executor() 2025-10-10T02:02:54.3437550Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:54.3438293Z (EngineCore_DP0 pid=13011) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:54.3439016Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:54.3446530Z (EngineCore_DP0 pid=13011) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:54.3447188Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3447989Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:54.3448631Z (EngineCore_DP0 pid=13011) return func(*args, **kwargs) 2025-10-10T02:02:54.3449048Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3449728Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:54.3450423Z (EngineCore_DP0 pid=13011) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:54.3450879Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3451571Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:54.3452248Z (EngineCore_DP0 pid=13011) module = importlib.import_module(module_name) 2025-10-10T02:02:54.3452702Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3453293Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:54.3453926Z (EngineCore_DP0 pid=13011) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:54.3454446Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3454960Z (EngineCore_DP0 pid=13011) File "", line 1387, in _gcd_import 2025-10-10T02:02:54.3455514Z (EngineCore_DP0 pid=13011) File "", line 1360, in _find_and_load 2025-10-10T02:02:54.3456141Z (EngineCore_DP0 pid=13011) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:54.3456736Z (EngineCore_DP0 pid=13011) File "", line 935, in _load_unlocked 2025-10-10T02:02:54.3457313Z (EngineCore_DP0 pid=13011) File "", line 999, in exec_module 2025-10-10T02:02:54.3457926Z (EngineCore_DP0 pid=13011) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:54.3458676Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:54.3459376Z (EngineCore_DP0 pid=13011) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:54.3460117Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:54.3460909Z (EngineCore_DP0 pid=13011) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:54.3461754Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:54.3462465Z (EngineCore_DP0 pid=13011) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:54.3463307Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:54.3464202Z (EngineCore_DP0 pid=13011) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:54.3464690Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3465402Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:54.3466098Z (EngineCore_DP0 pid=13011) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:54.3466592Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3467352Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:54.3468089Z (EngineCore_DP0 pid=13011) return _is_fa2_supported(device)[0] 2025-10-10T02:02:54.3468515Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3469249Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:54.3469984Z (EngineCore_DP0 pid=13011) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:54.3470442Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3471127Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:54.3471767Z (EngineCore_DP0 pid=13011) prop = get_device_properties(device) 2025-10-10T02:02:54.3472209Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:54.3472904Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:54.3473588Z (EngineCore_DP0 pid=13011) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:54.3473992Z (EngineCore_DP0 pid=13011) ^^^^^^^^^^^^ 2025-10-10T02:02:54.3474601Z (EngineCore_DP0 pid=13011) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:54.3475193Z (EngineCore_DP0 pid=13011) raise RuntimeError( 2025-10-10T02:02:54.3475908Z (EngineCore_DP0 pid=13011) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:54.7662743Z FAILED 2025-10-10T02:02:54.7793251Z models/test_initialization.py::test_implicit_converted_models[Qwen3ForSequenceClassification] Fork a new process to run a test 13015 2025-10-10T02:02:54.7804949Z Fork a new process to run a test 0 2025-10-10T02:02:54.8082799Z INFO 10-10 02:02:54 [utils.py:233] non-default args: {'load_format': 'dummy', 'gpu_memory_utilization': 0.8, 'disable_log_stats': True, 'hf_overrides': functools.partial(, model_arch='Qwen3ForSequenceClassification', exist_overrides={}, use_original_num_layers=False), 'model_impl': 'vllm', 'model': 'tomaarsen/Qwen3-Reranker-0.6B-seq-cls'} 2025-10-10T02:02:54.9075358Z 2025-10-10T02:02:54.9076412Z config.json: 0% 0.00/844 [00:00, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None} 2025-10-10T02:02:56.3849171Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] EngineCore failed to start. 2025-10-10T02:02:56.3849873Z 2025-10-10T02:02:56.3850527Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] Traceback (most recent call last): 2025-10-10T02:02:56.3850935Z 2025-10-10T02:02:56.3851886Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:56.3852540Z 2025-10-10T02:02:56.3852955Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:56.3853360Z 2025-10-10T02:02:56.3853679Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3853959Z 2025-10-10T02:02:56.3854453Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:56.3854899Z 2025-10-10T02:02:56.3855263Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:56.3855608Z 2025-10-10T02:02:56.3856120Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:56.3856568Z 2025-10-10T02:02:56.3856881Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:56.3857199Z 2025-10-10T02:02:56.3857491Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3857781Z 2025-10-10T02:02:56.3858299Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:56.3858792Z 2025-10-10T02:02:56.3859050Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] self._init_executor() 2025-10-10T02:02:56.3859324Z 2025-10-10T02:02:56.3859956Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:56.3860752Z 2025-10-10T02:02:56.3861439Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:56.3862073Z 2025-10-10T02:02:56.3863194Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:56.3863879Z 2025-10-10T02:02:56.3864302Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:56.3864784Z 2025-10-10T02:02:56.3865201Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3865508Z 2025-10-10T02:02:56.3866012Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:56.3866488Z 2025-10-10T02:02:56.3866858Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] return func(*args, **kwargs) 2025-10-10T02:02:56.3867147Z 2025-10-10T02:02:56.3867408Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3867680Z 2025-10-10T02:02:56.3868222Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:56.3868720Z 2025-10-10T02:02:56.3869021Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:56.3869370Z 2025-10-10T02:02:56.3869662Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3869944Z 2025-10-10T02:02:56.3870571Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:56.3871169Z 2025-10-10T02:02:56.3871508Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] module = importlib.import_module(module_name) 2025-10-10T02:02:56.3871824Z 2025-10-10T02:02:56.3872126Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3872406Z 2025-10-10T02:02:56.3872882Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:56.3873302Z 2025-10-10T02:02:56.3873664Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:56.3874006Z 2025-10-10T02:02:56.3874294Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3874580Z 2025-10-10T02:02:56.3874927Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "", line 1387, in _gcd_import 2025-10-10T02:02:56.3875270Z 2025-10-10T02:02:56.3875632Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "", line 1360, in _find_and_load 2025-10-10T02:02:56.3875977Z 2025-10-10T02:02:56.3876354Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:56.3876725Z 2025-10-10T02:02:56.3877077Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "", line 935, in _load_unlocked 2025-10-10T02:02:56.3877410Z 2025-10-10T02:02:56.3877788Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "", line 999, in exec_module 2025-10-10T02:02:56.3878141Z 2025-10-10T02:02:56.3878530Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:56.3878892Z 2025-10-10T02:02:56.3879588Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:56.3880133Z 2025-10-10T02:02:56.3880513Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:56.3880871Z 2025-10-10T02:02:56.3881611Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:56.3882223Z 2025-10-10T02:02:56.3882646Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:56.3883030Z 2025-10-10T02:02:56.3883604Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:56.3884142Z 2025-10-10T02:02:56.3884462Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] class FlashAttentionMetadataBuilder( 2025-10-10T02:02:56.3884770Z 2025-10-10T02:02:56.3885474Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:56.3886060Z 2025-10-10T02:02:56.3886446Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:56.3886800Z 2025-10-10T02:02:56.3887063Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3887325Z 2025-10-10T02:02:56.3887898Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:56.3888418Z 2025-10-10T02:02:56.3888715Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] if not is_fa_version_supported(fa_version): 2025-10-10T02:02:56.3889019Z 2025-10-10T02:02:56.3889292Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3889578Z 2025-10-10T02:02:56.3890182Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:56.3890748Z 2025-10-10T02:02:56.3891047Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] return _is_fa2_supported(device)[0] 2025-10-10T02:02:56.3891341Z 2025-10-10T02:02:56.3891607Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3891881Z 2025-10-10T02:02:56.3892486Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:56.3893018Z 2025-10-10T02:02:56.3893362Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:56.3893675Z 2025-10-10T02:02:56.3893973Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3894250Z 2025-10-10T02:02:56.3894788Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:56.3895288Z 2025-10-10T02:02:56.3895647Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] prop = get_device_properties(device) 2025-10-10T02:02:56.3895948Z 2025-10-10T02:02:56.3896506Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3896810Z 2025-10-10T02:02:56.3897363Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:56.3897867Z 2025-10-10T02:02:56.3898284Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] _lazy_init() # will define _get_device_properties 2025-10-10T02:02:56.3898596Z 2025-10-10T02:02:56.3898841Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] ^^^^^^^^^^^^ 2025-10-10T02:02:56.3899089Z 2025-10-10T02:02:56.3899579Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:56.3900036Z 2025-10-10T02:02:56.3900302Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] raise RuntimeError( 2025-10-10T02:02:56.3900563Z 2025-10-10T02:02:56.3901200Z (EngineCore_DP0 pid=13043) ERROR 10-10 02:02:56 [core.py:712] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:56.3901958Z (EngineCore_DP0 pid=13043) Process EngineCore_DP0: 2025-10-10T02:02:56.3902361Z (EngineCore_DP0 pid=13043) Traceback (most recent call last): 2025-10-10T02:02:56.3902979Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap 2025-10-10T02:02:56.3903523Z (EngineCore_DP0 pid=13043) self.run() 2025-10-10T02:02:56.3904059Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/multiprocessing/process.py", line 108, in run 2025-10-10T02:02:56.3904736Z (EngineCore_DP0 pid=13043) self._target(*self._args, **self._kwargs) 2025-10-10T02:02:56.3905608Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 716, in run_engine_core 2025-10-10T02:02:56.3906168Z (EngineCore_DP0 pid=13043) raise e 2025-10-10T02:02:56.3906771Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 703, in run_engine_core 2025-10-10T02:02:56.3907418Z (EngineCore_DP0 pid=13043) engine_core = EngineCoreProc(*args, **kwargs) 2025-10-10T02:02:56.3907882Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3908503Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 499, in __init__ 2025-10-10T02:02:56.3909165Z (EngineCore_DP0 pid=13043) super().__init__(vllm_config, executor_class, log_stats, 2025-10-10T02:02:56.3909840Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 84, in __init__ 2025-10-10T02:02:56.3910468Z (EngineCore_DP0 pid=13043) self.model_executor = executor_class(vllm_config) 2025-10-10T02:02:56.3910949Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3911615Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__ 2025-10-10T02:02:56.3912208Z (EngineCore_DP0 pid=13043) self._init_executor() 2025-10-10T02:02:56.3912887Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 53, in _init_executor 2025-10-10T02:02:56.3913702Z (EngineCore_DP0 pid=13043) self.collective_rpc("init_worker", args=([kwargs], )) 2025-10-10T02:02:56.3914529Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc 2025-10-10T02:02:56.3915245Z (EngineCore_DP0 pid=13043) return [run_method(self.driver_worker, method, args, kwargs)] 2025-10-10T02:02:56.3915758Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3916462Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3146, in run_method 2025-10-10T02:02:56.3917054Z (EngineCore_DP0 pid=13043) return func(*args, **kwargs) 2025-10-10T02:02:56.3917444Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3918083Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 254, in init_worker 2025-10-10T02:02:56.3918764Z (EngineCore_DP0 pid=13043) worker_class = resolve_obj_by_qualname( 2025-10-10T02:02:56.3919305Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3919999Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2704, in resolve_obj_by_qualname 2025-10-10T02:02:56.3920668Z (EngineCore_DP0 pid=13043) module = importlib.import_module(module_name) 2025-10-10T02:02:56.3921134Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3921714Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module 2025-10-10T02:02:56.3922342Z (EngineCore_DP0 pid=13043) return _bootstrap._gcd_import(name[level:], package, level) 2025-10-10T02:02:56.3922842Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3923355Z (EngineCore_DP0 pid=13043) File "", line 1387, in _gcd_import 2025-10-10T02:02:56.3923918Z (EngineCore_DP0 pid=13043) File "", line 1360, in _find_and_load 2025-10-10T02:02:56.3924515Z (EngineCore_DP0 pid=13043) File "", line 1331, in _find_and_load_unlocked 2025-10-10T02:02:56.3925114Z (EngineCore_DP0 pid=13043) File "", line 935, in _load_unlocked 2025-10-10T02:02:56.3925694Z (EngineCore_DP0 pid=13043) File "", line 999, in exec_module 2025-10-10T02:02:56.3926296Z (EngineCore_DP0 pid=13043) File "", line 488, in _call_with_frames_removed 2025-10-10T02:02:56.3927031Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 34, in 2025-10-10T02:02:56.3927720Z (EngineCore_DP0 pid=13043) from vllm.v1.worker.gpu_model_runner import GPUModelRunner 2025-10-10T02:02:56.3928454Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 69, in 2025-10-10T02:02:56.3929183Z (EngineCore_DP0 pid=13043) from vllm.v1.attention.backends.flash_attn import AttentionMetadata 2025-10-10T02:02:56.3929975Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 154, in 2025-10-10T02:02:56.3930665Z (EngineCore_DP0 pid=13043) class FlashAttentionMetadataBuilder( 2025-10-10T02:02:56.3931521Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 175, in FlashAttentionMetadataBuilder 2025-10-10T02:02:56.3932394Z (EngineCore_DP0 pid=13043) if get_flash_attn_version() == 3 else AttentionCGSupport.UNIFORM_BATCH 2025-10-10T02:02:56.3932905Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3933645Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/attention/utils/fa_utils.py", line 56, in get_flash_attn_version 2025-10-10T02:02:56.3934396Z (EngineCore_DP0 pid=13043) if not is_fa_version_supported(fa_version): 2025-10-10T02:02:56.3934856Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3935601Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 55, in is_fa_version_supported 2025-10-10T02:02:56.3936311Z (EngineCore_DP0 pid=13043) return _is_fa2_supported(device)[0] 2025-10-10T02:02:56.3936791Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3937526Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 35, in _is_fa2_supported 2025-10-10T02:02:56.3938264Z (EngineCore_DP0 pid=13043) if torch.cuda.get_device_capability(device)[0] < 8: 2025-10-10T02:02:56.3938727Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3939405Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 602, in get_device_capability 2025-10-10T02:02:56.3940058Z (EngineCore_DP0 pid=13043) prop = get_device_properties(device) 2025-10-10T02:02:56.3940492Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-10-10T02:02:56.3941169Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 619, in get_device_properties 2025-10-10T02:02:56.3941846Z (EngineCore_DP0 pid=13043) _lazy_init() # will define _get_device_properties 2025-10-10T02:02:56.3942254Z (EngineCore_DP0 pid=13043) ^^^^^^^^^^^^ 2025-10-10T02:02:56.3942846Z (EngineCore_DP0 pid=13043) File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py", line 399, in _lazy_init 2025-10-10T02:02:56.3943419Z (EngineCore_DP0 pid=13043) raise RuntimeError( 2025-10-10T02:02:56.3944123Z (EngineCore_DP0 pid=13043) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:56.8064605Z FAILED 2025-10-10T02:02:56.8064809Z 2025-10-10T02:02:56.8064985Z =================================== FAILURES =================================== 2025-10-10T02:02:56.8065609Z _______ test_can_initialize_small_subset[LlavaForConditionalGeneration] ________ 2025-10-10T02:02:56.8065965Z 2025-10-10T02:02:56.8066094Z model_arch = 'LlavaForConditionalGeneration' 2025-10-10T02:02:56.8066530Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212456660> 2025-10-10T02:02:56.8066840Z 2025-10-10T02:02:56.8294584Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8295501Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8296500Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8297390Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8298363Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8299594Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8300533Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8300852Z 2025-10-10T02:02:56.8301191Z models/test_initialization.py:118: 2025-10-10T02:02:56.8301537Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8302134Z utils.py:892: in wrapper 2025-10-10T02:02:56.8302562Z raise original_exception 2025-10-10T02:02:56.8302984Z utils.py:836: in wrapper 2025-10-10T02:02:56.8303542Z func(*args, **kwargs) 2025-10-10T02:02:56.8304087Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8304445Z LLM( 2025-10-10T02:02:56.8305265Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8305973Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8306736Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8307385Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8308061Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:56.8308731Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:56.8309338Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:56.8310004Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:56.8310735Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:56.8311407Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:56.8312152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:56.8312835Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:56.8313636Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:56.8314368Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:56.8315158Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:56.8315885Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:56.8316588Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:56.8317175Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:56.8317903Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:56.8318662Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:56.8319542Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:56.8320285Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:56.8321110Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:56.8321907Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:56.8322370Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8322553Z 2025-10-10T02:02:56.8322814Z > raise ValueError( 2025-10-10T02:02:56.8323461Z f"Model architectures {architectures} failed " 2025-10-10T02:02:56.8324171Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:56.8324929Z E ValueError: Model architectures ['LlavaForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:56.8325357Z 2025-10-10T02:02:56.8325777Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:56.8326429Z _______ test_can_initialize_small_subset[Llama4ForConditionalGeneration] _______ 2025-10-10T02:02:56.8326924Z 2025-10-10T02:02:56.8327104Z model_arch = 'Llama4ForConditionalGeneration' 2025-10-10T02:02:56.8327564Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22125259d0> 2025-10-10T02:02:56.8327867Z 2025-10-10T02:02:56.8328487Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8329282Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8329885Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8330529Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8331482Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8332179Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8333085Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8333341Z 2025-10-10T02:02:56.8333553Z models/test_initialization.py:118: 2025-10-10T02:02:56.8333974Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8334469Z utils.py:892: in wrapper 2025-10-10T02:02:56.8334910Z raise original_exception 2025-10-10T02:02:56.8335428Z utils.py:836: in wrapper 2025-10-10T02:02:56.8335882Z func(*args, **kwargs) 2025-10-10T02:02:56.8336352Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8336725Z LLM( 2025-10-10T02:02:56.8337451Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8338156Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8338947Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8339678Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8340464Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8341236Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8342104Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8342810Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8343413Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8344053Z super().__init__( 2025-10-10T02:02:56.8344723Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8345303Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8345900Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8346475Z next(self.gen) 2025-10-10T02:02:56.8347036Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8347592Z wait_for_engine_startup( 2025-10-10T02:02:56.8347850Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8348024Z 2025-10-10T02:02:56.8348447Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8349066Z "See root cause above. " 2025-10-10T02:02:56.8349659Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8350365Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8350744Z 2025-10-10T02:02:56.8351080Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8352095Z _______ test_can_initialize_small_subset[BertForSequenceClassification] ________ 2025-10-10T02:02:56.8352486Z 2025-10-10T02:02:56.8352600Z model_arch = 'BertForSequenceClassification' 2025-10-10T02:02:56.8352939Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22125d3980> 2025-10-10T02:02:56.8353258Z 2025-10-10T02:02:56.8353744Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8354423Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8355111Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8355771Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8356381Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8357043Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8357756Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8358007Z 2025-10-10T02:02:56.8358160Z models/test_initialization.py:118: 2025-10-10T02:02:56.8358427Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8358908Z utils.py:892: in wrapper 2025-10-10T02:02:56.8359422Z raise original_exception 2025-10-10T02:02:56.8359727Z utils.py:836: in wrapper 2025-10-10T02:02:56.8360072Z func(*args, **kwargs) 2025-10-10T02:02:56.8360666Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8361083Z LLM( 2025-10-10T02:02:56.8361541Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8362288Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8363170Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8363924Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8364659Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8365338Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8366115Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8366962Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8367590Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8368094Z super().__init__( 2025-10-10T02:02:56.8368613Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8369329Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8370099Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8370681Z next(self.gen) 2025-10-10T02:02:56.8371226Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8371706Z wait_for_engine_startup( 2025-10-10T02:02:56.8371962Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8372132Z 2025-10-10T02:02:56.8372547Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8373161Z "See root cause above. " 2025-10-10T02:02:56.8373753Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8374457Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8374833Z 2025-10-10T02:02:56.8375158Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8375912Z _____________ test_can_initialize_small_subset[Gemma3nForCausalLM] _____________ 2025-10-10T02:02:56.8376319Z 2025-10-10T02:02:56.8376406Z model_arch = 'Gemma3nForCausalLM' 2025-10-10T02:02:56.8376731Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22125259a0> 2025-10-10T02:02:56.8376974Z 2025-10-10T02:02:56.8377374Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8378239Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8378744Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8379439Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8380263Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8380930Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8381495Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8381978Z 2025-10-10T02:02:56.8382145Z models/test_initialization.py:118: 2025-10-10T02:02:56.8382416Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8382758Z utils.py:892: in wrapper 2025-10-10T02:02:56.8383187Z raise original_exception 2025-10-10T02:02:56.8383492Z utils.py:836: in wrapper 2025-10-10T02:02:56.8383751Z func(*args, **kwargs) 2025-10-10T02:02:56.8384083Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8384436Z LLM( 2025-10-10T02:02:56.8385092Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8385650Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8386412Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8387127Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8387756Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8388352Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8389109Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8389880Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8390646Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8391173Z super().__init__( 2025-10-10T02:02:56.8391706Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8392346Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8392963Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8393383Z next(self.gen) 2025-10-10T02:02:56.8394079Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8394612Z wait_for_engine_startup( 2025-10-10T02:02:56.8394867Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8395038Z 2025-10-10T02:02:56.8395458Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8396069Z "See root cause above. " 2025-10-10T02:02:56.8396861Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8397570Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8397939Z 2025-10-10T02:02:56.8398383Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8399023Z ______________ test_can_initialize_small_subset[JinaVLForRanking] ______________ 2025-10-10T02:02:56.8399377Z 2025-10-10T02:02:56.8399457Z model_arch = 'JinaVLForRanking' 2025-10-10T02:02:56.8399900Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a9565d0> 2025-10-10T02:02:56.8400242Z 2025-10-10T02:02:56.8400815Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8401763Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8402317Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8402801Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8403579Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8404478Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8405063Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8405339Z 2025-10-10T02:02:56.8405618Z models/test_initialization.py:118: 2025-10-10T02:02:56.8405912Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8406213Z utils.py:892: in wrapper 2025-10-10T02:02:56.8406529Z raise original_exception 2025-10-10T02:02:56.8407010Z utils.py:836: in wrapper 2025-10-10T02:02:56.8407287Z func(*args, **kwargs) 2025-10-10T02:02:56.8407615Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8407925Z LLM( 2025-10-10T02:02:56.8408503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8409114Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8409712Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8410504Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8411157Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8411818Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8412388Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8413187Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8413842Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8414498Z super().__init__( 2025-10-10T02:02:56.8415193Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8415781Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8416248Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8416767Z next(self.gen) 2025-10-10T02:02:56.8417475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8418040Z wait_for_engine_startup( 2025-10-10T02:02:56.8418404Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8418689Z 2025-10-10T02:02:56.8419132Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8419753Z "See root cause above. " 2025-10-10T02:02:56.8420413Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8421232Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8421605Z 2025-10-10T02:02:56.8421924Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8422497Z _____________ test_can_initialize_small_subset[InternVLChatModel] ______________ 2025-10-10T02:02:56.8422753Z 2025-10-10T02:02:56.8422892Z model_arch = 'InternVLChatModel' 2025-10-10T02:02:56.8423193Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212526d80> 2025-10-10T02:02:56.8423440Z 2025-10-10T02:02:56.8424879Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8425584Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8426268Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8426758Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8427340Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8428154Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8428890Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8429139Z 2025-10-10T02:02:56.8429291Z models/test_initialization.py:118: 2025-10-10T02:02:56.8429564Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8429901Z utils.py:892: in wrapper 2025-10-10T02:02:56.8430346Z raise original_exception 2025-10-10T02:02:56.8430667Z utils.py:836: in wrapper 2025-10-10T02:02:56.8430929Z func(*args, **kwargs) 2025-10-10T02:02:56.8431295Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8431737Z LLM( 2025-10-10T02:02:56.8432198Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8432801Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8433514Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8434102Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8434839Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8435424Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8436186Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8436865Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8437741Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8438389Z super().__init__( 2025-10-10T02:02:56.8439004Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8439798Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8440356Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8440784Z next(self.gen) 2025-10-10T02:02:56.8441610Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8442117Z wait_for_engine_startup( 2025-10-10T02:02:56.8442375Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8442559Z 2025-10-10T02:02:56.8443211Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8443853Z "See root cause above. " 2025-10-10T02:02:56.8444452Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8445149Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8445525Z 2025-10-10T02:02:56.8445855Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8446449Z __________ test_can_initialize_small_subset[InternLM2ForRewardModel] ___________ 2025-10-10T02:02:56.8446719Z 2025-10-10T02:02:56.8446807Z model_arch = 'InternLM2ForRewardModel' 2025-10-10T02:02:56.8447132Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a956030> 2025-10-10T02:02:56.8447371Z 2025-10-10T02:02:56.8449283Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8450130Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8450735Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8451212Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8451967Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8452765Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8453338Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8453760Z 2025-10-10T02:02:56.8453939Z models/test_initialization.py:118: 2025-10-10T02:02:56.8454223Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8454526Z utils.py:892: in wrapper 2025-10-10T02:02:56.8455022Z raise original_exception 2025-10-10T02:02:56.8455342Z utils.py:836: in wrapper 2025-10-10T02:02:56.8455614Z func(*args, **kwargs) 2025-10-10T02:02:56.8456102Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8456431Z LLM( 2025-10-10T02:02:56.8456956Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8457570Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8458332Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8459121Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8459729Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8460563Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8461307Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8461941Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8462745Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8463277Z super().__init__( 2025-10-10T02:02:56.8463877Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8464679Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8465189Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8465789Z next(self.gen) 2025-10-10T02:02:56.8466351Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8466833Z wait_for_engine_startup( 2025-10-10T02:02:56.8467084Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8467266Z 2025-10-10T02:02:56.8467675Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8468298Z "See root cause above. " 2025-10-10T02:02:56.8468891Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8469590Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8469961Z 2025-10-10T02:02:56.8470280Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8470851Z ________ test_can_initialize_small_subset[TransformersForMultimodalLM] _________ 2025-10-10T02:02:56.8471224Z 2025-10-10T02:02:56.8471405Z model_arch = 'TransformersForMultimodalLM' 2025-10-10T02:02:56.8471811Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22125d1280> 2025-10-10T02:02:56.8472052Z 2025-10-10T02:02:56.8472459Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8473303Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8473823Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8474387Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8475190Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8475906Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8476532Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8476948Z 2025-10-10T02:02:56.8477123Z models/test_initialization.py:118: 2025-10-10T02:02:56.8477388Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8477754Z utils.py:892: in wrapper 2025-10-10T02:02:56.8478262Z raise original_exception 2025-10-10T02:02:56.8478575Z utils.py:836: in wrapper 2025-10-10T02:02:56.8478832Z func(*args, **kwargs) 2025-10-10T02:02:56.8479275Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8479773Z LLM( 2025-10-10T02:02:56.8480246Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8480787Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8481625Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8482422Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8483008Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8483747Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8484362Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8485142Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8485909Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8486429Z super().__init__( 2025-10-10T02:02:56.8486965Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8487556Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8488230Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8488662Z next(self.gen) 2025-10-10T02:02:56.8489365Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8489873Z wait_for_engine_startup( 2025-10-10T02:02:56.8490129Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8490304Z 2025-10-10T02:02:56.8490730Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8491331Z "See root cause above. " 2025-10-10T02:02:56.8491926Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8492623Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8492991Z 2025-10-10T02:02:56.8493321Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8493891Z ____________ test_can_initialize_small_subset[PrithviGeoSpatialMAE] ____________ 2025-10-10T02:02:56.8494157Z 2025-10-10T02:02:56.8494245Z model_arch = 'PrithviGeoSpatialMAE' 2025-10-10T02:02:56.8494728Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a955f70> 2025-10-10T02:02:56.8495234Z 2025-10-10T02:02:56.8495654Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8496684Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8497221Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8497830Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8498645Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8499388Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8500015Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8500353Z 2025-10-10T02:02:56.8500607Z models/test_initialization.py:118: 2025-10-10T02:02:56.8500878Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8501359Z utils.py:892: in wrapper 2025-10-10T02:02:56.8501707Z raise original_exception 2025-10-10T02:02:56.8502020Z utils.py:836: in wrapper 2025-10-10T02:02:56.8502422Z func(*args, **kwargs) 2025-10-10T02:02:56.8502816Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8503133Z LLM( 2025-10-10T02:02:56.8503594Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8504311Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8505087Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8505687Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8506380Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8506986Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8507761Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8508532Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8509147Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8509664Z super().__init__( 2025-10-10T02:02:56.8510222Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8510945Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8511420Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8512013Z next(self.gen) 2025-10-10T02:02:56.8512592Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8513081Z wait_for_engine_startup( 2025-10-10T02:02:56.8513343Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8513517Z 2025-10-10T02:02:56.8514044Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8514727Z "See root cause above. " 2025-10-10T02:02:56.8515312Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8515950Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} 2025-10-10T02:02:56.8516330Z 2025-10-10T02:02:56.8516649Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8517360Z _______________ test_can_initialize_small_subset[UltravoxModel] ________________ 2025-10-10T02:02:56.8517724Z 2025-10-10T02:02:56.8517812Z model_arch = 'UltravoxModel' 2025-10-10T02:02:56.8518188Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a954f80> 2025-10-10T02:02:56.8518506Z 2025-10-10T02:02:56.8519210Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8519934Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8520432Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8521122Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8521874Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8522531Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8523266Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8523525Z 2025-10-10T02:02:56.8523684Z models/test_initialization.py:118: 2025-10-10T02:02:56.8524094Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8524399Z utils.py:892: in wrapper 2025-10-10T02:02:56.8524715Z raise original_exception 2025-10-10T02:02:56.8525080Z utils.py:836: in wrapper 2025-10-10T02:02:56.8525464Z func(*args, **kwargs) 2025-10-10T02:02:56.8525804Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8526107Z LLM( 2025-10-10T02:02:56.8526663Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8527257Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8528017Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8528623Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8529343Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8529935Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8530708Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8531512Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8532152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8532670Z super().__init__( 2025-10-10T02:02:56.8533550Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8534286Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8534900Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8535323Z next(self.gen) 2025-10-10T02:02:56.8535861Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8536416Z wait_for_engine_startup( 2025-10-10T02:02:56.8536673Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8536848Z 2025-10-10T02:02:56.8537271Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8537895Z "See root cause above. " 2025-10-10T02:02:56.8538539Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8539275Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8539900Z 2025-10-10T02:02:56.8540338Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8540972Z ______________ test_can_initialize_small_subset[DeepSeekMTPModel] ______________ 2025-10-10T02:02:56.8541350Z 2025-10-10T02:02:56.8541447Z model_arch = 'DeepSeekMTPModel' 2025-10-10T02:02:56.8541758Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212525c40> 2025-10-10T02:02:56.8542001Z 2025-10-10T02:02:56.8542414Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8543181Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8543762Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8544412Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8545015Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8545652Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8546372Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8546627Z 2025-10-10T02:02:56.8546790Z models/test_initialization.py:118: 2025-10-10T02:02:56.8547054Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8547385Z utils.py:892: in wrapper 2025-10-10T02:02:56.8547836Z raise original_exception 2025-10-10T02:02:56.8548134Z utils.py:836: in wrapper 2025-10-10T02:02:56.8548400Z func(*args, **kwargs) 2025-10-10T02:02:56.8548734Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8549034Z LLM( 2025-10-10T02:02:56.8549506Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8550181Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8550771Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8551492Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8552199Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8552864Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8553530Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8554201Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8554895Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8555534Z super().__init__( 2025-10-10T02:02:56.8556214Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8556825Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8557553Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8558024Z next(self.gen) 2025-10-10T02:02:56.8558578Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8559168Z wait_for_engine_startup( 2025-10-10T02:02:56.8559430Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8559609Z 2025-10-10T02:02:56.8560761Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8561556Z "See root cause above. " 2025-10-10T02:02:56.8562208Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8562927Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8563301Z 2025-10-10T02:02:56.8563649Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8564216Z ______________ test_can_initialize_small_subset[XLMRobertaModel] _______________ 2025-10-10T02:02:56.8564480Z 2025-10-10T02:02:56.8564560Z model_arch = 'XLMRobertaModel' 2025-10-10T02:02:56.8564866Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22125d1280> 2025-10-10T02:02:56.8565106Z 2025-10-10T02:02:56.8568590Z @pytest.mark.parametrize("model_arch", MINIMAL_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8569484Z def test_can_initialize_small_subset(model_arch: str, 2025-10-10T02:02:56.8570001Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8570482Z  """Test initializing small subset of supported models""" 2025-10-10T02:02:56.8571155Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8572008Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8572617Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8572869Z 2025-10-10T02:02:56.8573013Z models/test_initialization.py:118: 2025-10-10T02:02:56.8573352Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8573704Z utils.py:892: in wrapper 2025-10-10T02:02:56.8574225Z raise original_exception 2025-10-10T02:02:56.8574552Z utils.py:836: in wrapper 2025-10-10T02:02:56.8574822Z func(*args, **kwargs) 2025-10-10T02:02:56.8575174Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8575737Z LLM( 2025-10-10T02:02:56.8576187Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8576723Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8577396Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8578065Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8578635Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8579405Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8580060Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8580837Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8581478Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8582006Z super().__init__( 2025-10-10T02:02:56.8582702Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8583306Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8583984Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8584425Z next(self.gen) 2025-10-10T02:02:56.8585122Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8585665Z wait_for_engine_startup( 2025-10-10T02:02:56.8585933Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8586109Z 2025-10-10T02:02:56.8588565Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8589630Z "See root cause above. " 2025-10-10T02:02:56.8590262Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8590978Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8591357Z 2025-10-10T02:02:56.8591687Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8592274Z ______________ test_can_initialize_large_subset[Olmo2ForCausalLM] ______________ 2025-10-10T02:02:56.8592528Z 2025-10-10T02:02:56.8592614Z model_arch = 'Olmo2ForCausalLM' 2025-10-10T02:02:56.8592913Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212526e40> 2025-10-10T02:02:56.8593162Z 2025-10-10T02:02:56.8605824Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8606926Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8607742Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8608218Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8608707Z  2025-10-10T02:02:56.8609220Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8609938Z  test. 2025-10-10T02:02:56.8610278Z  """ 2025-10-10T02:02:56.8610732Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8611376Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8612072Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8612464Z 2025-10-10T02:02:56.8612734Z models/test_initialization.py:131: 2025-10-10T02:02:56.8613168Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8613646Z utils.py:892: in wrapper 2025-10-10T02:02:56.8614222Z raise original_exception 2025-10-10T02:02:56.8614770Z utils.py:836: in wrapper 2025-10-10T02:02:56.8615072Z func(*args, **kwargs) 2025-10-10T02:02:56.8615408Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8615706Z LLM( 2025-10-10T02:02:56.8616162Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8616699Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8617284Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8617868Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8618431Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8619000Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8619605Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8620323Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8621244Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8621823Z super().__init__( 2025-10-10T02:02:56.8622522Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8623123Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8623616Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8624259Z next(self.gen) 2025-10-10T02:02:56.8624832Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8625415Z wait_for_engine_startup( 2025-10-10T02:02:56.8625680Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8625857Z 2025-10-10T02:02:56.8626278Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8627046Z "See root cause above. " 2025-10-10T02:02:56.8627726Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8628557Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8628998Z 2025-10-10T02:02:56.8629339Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8629919Z _______________ test_can_initialize_large_subset[ErnieMTPModel] ________________ 2025-10-10T02:02:56.8630207Z 2025-10-10T02:02:56.8630351Z model_arch = 'ErnieMTPModel' 2025-10-10T02:02:56.8630665Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212526ab0> 2025-10-10T02:02:56.8630906Z 2025-10-10T02:02:56.8631285Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8632039Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8632604Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8633047Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8633480Z  2025-10-10T02:02:56.8633863Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8634242Z  test. 2025-10-10T02:02:56.8634486Z  """ 2025-10-10T02:02:56.8634932Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8635670Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8636234Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8636574Z 2025-10-10T02:02:56.8636725Z models/test_initialization.py:131: 2025-10-10T02:02:56.8637000Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8637298Z utils.py:892: in wrapper 2025-10-10T02:02:56.8637703Z raise original_exception 2025-10-10T02:02:56.8638019Z utils.py:836: in wrapper 2025-10-10T02:02:56.8638306Z func(*args, **kwargs) 2025-10-10T02:02:56.8638644Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8638950Z LLM( 2025-10-10T02:02:56.8639509Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8640048Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8640768Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8641366Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8642118Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8642699Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8643288Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8643893Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8644564Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8645139Z super().__init__( 2025-10-10T02:02:56.8645674Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8646245Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8646790Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8647203Z next(self.gen) 2025-10-10T02:02:56.8647740Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8648357Z wait_for_engine_startup( 2025-10-10T02:02:56.8648713Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8648966Z 2025-10-10T02:02:56.8649469Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8650213Z "See root cause above. " 2025-10-10T02:02:56.8650812Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8651530Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8651912Z 2025-10-10T02:02:56.8652335Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8653016Z _____________ test_can_initialize_large_subset[GptOssForCausalLM] ______________ 2025-10-10T02:02:56.8653279Z 2025-10-10T02:02:56.8653362Z model_arch = 'GptOssForCausalLM' 2025-10-10T02:02:56.8653675Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249ade0> 2025-10-10T02:02:56.8653916Z 2025-10-10T02:02:56.8654316Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8655073Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8655589Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8656022Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8656452Z  2025-10-10T02:02:56.8656832Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8657200Z  test. 2025-10-10T02:02:56.8657462Z  """ 2025-10-10T02:02:56.8657927Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8658669Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8659245Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8659494Z 2025-10-10T02:02:56.8659633Z models/test_initialization.py:131: 2025-10-10T02:02:56.8659911Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8660298Z utils.py:892: in wrapper 2025-10-10T02:02:56.8660620Z raise original_exception 2025-10-10T02:02:56.8660915Z utils.py:836: in wrapper 2025-10-10T02:02:56.8661175Z func(*args, **kwargs) 2025-10-10T02:02:56.8661669Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8662031Z LLM( 2025-10-10T02:02:56.8662473Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8663017Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8663739Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8664561Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8665189Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8665984Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8666586Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8667193Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8667865Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8668612Z super().__init__( 2025-10-10T02:02:56.8669164Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8670077Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8670597Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8671023Z next(self.gen) 2025-10-10T02:02:56.8671578Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8672073Z wait_for_engine_startup( 2025-10-10T02:02:56.8672337Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8672515Z 2025-10-10T02:02:56.8672934Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8673549Z "See root cause above. " 2025-10-10T02:02:56.8674144Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8674838Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8675222Z 2025-10-10T02:02:56.8675540Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8676430Z ______ test_can_initialize_large_subset[SmolVLMForConditionalGeneration] _______ 2025-10-10T02:02:56.8676961Z 2025-10-10T02:02:56.8677084Z model_arch = 'SmolVLMForConditionalGeneration' 2025-10-10T02:02:56.8677584Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124f5f40> 2025-10-10T02:02:56.8677833Z 2025-10-10T02:02:56.8678242Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8679167Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8679681Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8680121Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8680544Z  2025-10-10T02:02:56.8681154Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8681690Z  test. 2025-10-10T02:02:56.8681945Z  """ 2025-10-10T02:02:56.8682383Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8683024Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8683843Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8684105Z 2025-10-10T02:02:56.8684255Z models/test_initialization.py:131: 2025-10-10T02:02:56.8684525Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8684983Z utils.py:892: in wrapper 2025-10-10T02:02:56.8685330Z raise original_exception 2025-10-10T02:02:56.8685635Z utils.py:836: in wrapper 2025-10-10T02:02:56.8685967Z func(*args, **kwargs) 2025-10-10T02:02:56.8686318Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8686631Z LLM( 2025-10-10T02:02:56.8687270Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8687838Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8688710Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8689511Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8690390Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8691014Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8691809Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8692813Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8693720Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8694539Z super().__init__( 2025-10-10T02:02:56.8695091Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8695675Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8696371Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8697016Z next(self.gen) 2025-10-10T02:02:56.8697866Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8698403Z wait_for_engine_startup( 2025-10-10T02:02:56.8698815Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8699003Z 2025-10-10T02:02:56.8699439Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8700061Z "See root cause above. " 2025-10-10T02:02:56.8700655Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8701463Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8701900Z 2025-10-10T02:02:56.8702242Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8702830Z _____ test_can_initialize_large_subset[LlavaNextForConditionalGeneration] ______ 2025-10-10T02:02:56.8703115Z 2025-10-10T02:02:56.8703228Z model_arch = 'LlavaNextForConditionalGeneration' 2025-10-10T02:02:56.8703643Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a956840> 2025-10-10T02:02:56.8703882Z 2025-10-10T02:02:56.8706015Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8706902Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8707441Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8707881Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8708310Z  2025-10-10T02:02:56.8708899Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8709426Z  test. 2025-10-10T02:02:56.8709679Z  """ 2025-10-10T02:02:56.8710142Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8710969Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8711554Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8711936Z 2025-10-10T02:02:56.8712125Z models/test_initialization.py:131: 2025-10-10T02:02:56.8712407Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8712703Z utils.py:892: in wrapper 2025-10-10T02:02:56.8713096Z raise original_exception 2025-10-10T02:02:56.8713510Z utils.py:836: in wrapper 2025-10-10T02:02:56.8713786Z func(*args, **kwargs) 2025-10-10T02:02:56.8714178Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8714619Z LLM( 2025-10-10T02:02:56.8715078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8715613Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8716405Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8717014Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8717766Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:56.8718359Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:56.8718944Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:56.8719851Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:56.8720498Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:56.8721355Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:56.8722094Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:56.8723023Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:56.8723897Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:56.8724645Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:56.8725504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:56.8726231Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:56.8726998Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:56.8727562Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:56.8728205Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:56.8729146Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:56.8729883Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:56.8730743Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:56.8731401Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:56.8732051Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:56.8732418Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8732598Z 2025-10-10T02:02:56.8732784Z > raise ValueError( 2025-10-10T02:02:56.8733420Z f"Model architectures {architectures} failed " 2025-10-10T02:02:56.8734143Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:56.8734915Z E ValueError: Model architectures ['LlavaNextForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:56.8735365Z 2025-10-10T02:02:56.8735728Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:56.8736713Z ____________ test_can_initialize_large_subset[Step3TextForCausalLM] ____________ 2025-10-10T02:02:56.8736982Z 2025-10-10T02:02:56.8737067Z model_arch = 'Step3TextForCausalLM' 2025-10-10T02:02:56.8737388Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124f7d10> 2025-10-10T02:02:56.8737630Z 2025-10-10T02:02:56.8738201Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8738875Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8739530Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8740123Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8740469Z  2025-10-10T02:02:56.8740845Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8741221Z  test. 2025-10-10T02:02:56.8741471Z  """ 2025-10-10T02:02:56.8742723Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8743563Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8744144Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8744552Z 2025-10-10T02:02:56.8744714Z models/test_initialization.py:131: 2025-10-10T02:02:56.8745048Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8745347Z utils.py:892: in wrapper 2025-10-10T02:02:56.8745722Z raise original_exception 2025-10-10T02:02:56.8746161Z utils.py:836: in wrapper 2025-10-10T02:02:56.8746433Z func(*args, **kwargs) 2025-10-10T02:02:56.8746793Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8747277Z LLM( 2025-10-10T02:02:56.8747735Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8748475Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8749171Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8749884Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8750642Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8751216Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8751789Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8752591Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8753208Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8753935Z super().__init__( 2025-10-10T02:02:56.8754607Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8755191Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8755670Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8756093Z next(self.gen) 2025-10-10T02:02:56.8756629Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8757111Z wait_for_engine_startup( 2025-10-10T02:02:56.8757363Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8757541Z 2025-10-10T02:02:56.8757966Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8758565Z "See root cause above. " 2025-10-10T02:02:56.8759243Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8760408Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8760801Z 2025-10-10T02:02:56.8761154Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8761968Z ____________ test_can_initialize_large_subset[BaichuanForCausalLM] _____________ 2025-10-10T02:02:56.8762280Z 2025-10-10T02:02:56.8762366Z model_arch = 'BaichuanForCausalLM' 2025-10-10T02:02:56.8762686Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22125d2b40> 2025-10-10T02:02:56.8762932Z 2025-10-10T02:02:56.8763516Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8764419Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8764920Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8765387Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8765865Z  2025-10-10T02:02:56.8766240Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8766685Z  test. 2025-10-10T02:02:56.8767020Z  """ 2025-10-10T02:02:56.8767524Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8768203Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8768915Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8769156Z 2025-10-10T02:02:56.8769307Z models/test_initialization.py:131: 2025-10-10T02:02:56.8769564Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8770007Z utils.py:892: in wrapper 2025-10-10T02:02:56.8770361Z raise original_exception 2025-10-10T02:02:56.8770687Z utils.py:836: in wrapper 2025-10-10T02:02:56.8771107Z func(*args, **kwargs) 2025-10-10T02:02:56.8771447Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8771754Z LLM( 2025-10-10T02:02:56.8772244Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8772912Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8773680Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8774408Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8774985Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8775540Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8776114Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8776929Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8777559Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8778211Z super().__init__( 2025-10-10T02:02:56.8778764Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8779335Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8779807Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8780211Z next(self.gen) 2025-10-10T02:02:56.8780809Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8781334Z wait_for_engine_startup( 2025-10-10T02:02:56.8781597Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8781772Z 2025-10-10T02:02:56.8782179Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8782830Z "See root cause above. " 2025-10-10T02:02:56.8783683Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8784561Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8784970Z 2025-10-10T02:02:56.8785439Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8786109Z ______ test_can_initialize_large_subset[TarsierForConditionalGeneration] _______ 2025-10-10T02:02:56.8786393Z 2025-10-10T02:02:56.8786504Z model_arch = 'TarsierForConditionalGeneration' 2025-10-10T02:02:56.8787006Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124f6c60> 2025-10-10T02:02:56.8787262Z 2025-10-10T02:02:56.8787814Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8788488Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8789025Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8789587Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8789924Z  2025-10-10T02:02:56.8790434Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8790854Z  test. 2025-10-10T02:02:56.8791098Z  """ 2025-10-10T02:02:56.8791538Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8792344Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8793065Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8793318Z 2025-10-10T02:02:56.8793469Z models/test_initialization.py:131: 2025-10-10T02:02:56.8793735Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8794099Z utils.py:892: in wrapper 2025-10-10T02:02:56.8794500Z raise original_exception 2025-10-10T02:02:56.8794808Z utils.py:836: in wrapper 2025-10-10T02:02:56.8795069Z func(*args, **kwargs) 2025-10-10T02:02:56.8795571Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8795895Z LLM( 2025-10-10T02:02:56.8796566Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8797184Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8797898Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8798653Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8799413Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:56.8800091Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:56.8800872Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:56.8801612Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:56.8802378Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:56.8803031Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:56.8803907Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:56.8804760Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:56.8805553Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:56.8806400Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:56.8807218Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:56.8807997Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:56.8808585Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:56.8809128Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:56.8809758Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:56.8810363Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:56.8810999Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:56.8811622Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:56.8812265Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:56.8812910Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:56.8813273Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8813549Z 2025-10-10T02:02:56.8813913Z > raise ValueError( 2025-10-10T02:02:56.8814641Z f"Model architectures {architectures} failed " 2025-10-10T02:02:56.8815512Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:56.8816331Z E ValueError: Model architectures ['TarsierForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:56.8816787Z 2025-10-10T02:02:56.8817349Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:56.8818140Z ___________ test_can_initialize_large_subset[Eagle3LlamaForCausalLM] ___________ 2025-10-10T02:02:56.8818402Z 2025-10-10T02:02:56.8818499Z model_arch = 'Eagle3LlamaForCausalLM' 2025-10-10T02:02:56.8818825Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a957b90> 2025-10-10T02:02:56.8819159Z 2025-10-10T02:02:56.8819787Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8820586Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8821176Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8821683Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8822191Z  2025-10-10T02:02:56.8822583Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8822972Z  test. 2025-10-10T02:02:56.8823212Z  """ 2025-10-10T02:02:56.8823832Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8824629Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8825269Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8825523Z 2025-10-10T02:02:56.8825673Z models/test_initialization.py:131: 2025-10-10T02:02:56.8825997Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8826428Z utils.py:892: in wrapper 2025-10-10T02:02:56.8826742Z raise original_exception 2025-10-10T02:02:56.8827131Z utils.py:836: in wrapper 2025-10-10T02:02:56.8827500Z func(*args, **kwargs) 2025-10-10T02:02:56.8827874Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8828315Z LLM( 2025-10-10T02:02:56.8828768Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8829315Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8829961Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8830770Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8831342Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8832086Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8832679Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8833292Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8833902Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8834422Z super().__init__( 2025-10-10T02:02:56.8834944Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8835509Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8835983Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8836401Z next(self.gen) 2025-10-10T02:02:56.8836953Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8837780Z wait_for_engine_startup( 2025-10-10T02:02:56.8838247Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8838475Z 2025-10-10T02:02:56.8838970Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8839846Z "See root cause above. " 2025-10-10T02:02:56.8840447Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8841521Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8841958Z 2025-10-10T02:02:56.8842303Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8842887Z _____________ test_can_initialize_large_subset[CohereForCausalLM] ______________ 2025-10-10T02:02:56.8843319Z 2025-10-10T02:02:56.8843416Z model_arch = 'CohereForCausalLM' 2025-10-10T02:02:56.8843733Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b1a00> 2025-10-10T02:02:56.8844130Z 2025-10-10T02:02:56.8844659Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8845332Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8845970Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8846440Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8846783Z  2025-10-10T02:02:56.8847153Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8847693Z  test. 2025-10-10T02:02:56.8847946Z  """ 2025-10-10T02:02:56.8848489Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8849196Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8849875Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8850164Z 2025-10-10T02:02:56.8850319Z models/test_initialization.py:131: 2025-10-10T02:02:56.8850583Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8851011Z utils.py:892: in wrapper 2025-10-10T02:02:56.8851379Z raise original_exception 2025-10-10T02:02:56.8851821Z utils.py:836: in wrapper 2025-10-10T02:02:56.8852116Z func(*args, **kwargs) 2025-10-10T02:02:56.8852460Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8852768Z LLM( 2025-10-10T02:02:56.8853220Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8853900Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8854526Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8855239Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8855865Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8856425Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8857068Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8857724Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8858330Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8858845Z super().__init__( 2025-10-10T02:02:56.8859367Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8859975Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8860456Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8861054Z next(self.gen) 2025-10-10T02:02:56.8861904Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8862432Z wait_for_engine_startup( 2025-10-10T02:02:56.8862850Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8863088Z 2025-10-10T02:02:56.8863525Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8864195Z "See root cause above. " 2025-10-10T02:02:56.8865040Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8865797Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8866169Z 2025-10-10T02:02:56.8866527Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8867256Z ____________ test_can_initialize_large_subset[MiniMaxM1ForCausalLM] ____________ 2025-10-10T02:02:56.8867518Z 2025-10-10T02:02:56.8867604Z model_arch = 'MiniMaxM1ForCausalLM' 2025-10-10T02:02:56.8868084Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249af00> 2025-10-10T02:02:56.8868325Z 2025-10-10T02:02:56.8868709Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8869413Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8870050Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8870477Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8870810Z  2025-10-10T02:02:56.8871352Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8871755Z  test. 2025-10-10T02:02:56.8872034Z  """ 2025-10-10T02:02:56.8872592Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8873224Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8873968Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8874210Z 2025-10-10T02:02:56.8874364Z models/test_initialization.py:131: 2025-10-10T02:02:56.8874788Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8875102Z utils.py:892: in wrapper 2025-10-10T02:02:56.8875631Z raise original_exception 2025-10-10T02:02:56.8875995Z utils.py:836: in wrapper 2025-10-10T02:02:56.8876260Z func(*args, **kwargs) 2025-10-10T02:02:56.8876588Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8876897Z LLM( 2025-10-10T02:02:56.8877361Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8878138Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8878814Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8879605Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8880186Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8880742Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8881403Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8882009Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8882614Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8883118Z super().__init__( 2025-10-10T02:02:56.8883633Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8884239Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8885059Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8885512Z next(self.gen) 2025-10-10T02:02:56.8886251Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8886779Z wait_for_engine_startup( 2025-10-10T02:02:56.8887105Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8887403Z 2025-10-10T02:02:56.8887986Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8888621Z "See root cause above. " 2025-10-10T02:02:56.8889220Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8890130Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8890568Z 2025-10-10T02:02:56.8891027Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8891624Z ______________ test_can_initialize_large_subset[OlmoeForCausalLM] ______________ 2025-10-10T02:02:56.8891878Z 2025-10-10T02:02:56.8891958Z model_arch = 'OlmoeForCausalLM' 2025-10-10T02:02:56.8892428Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124f78c0> 2025-10-10T02:02:56.8892684Z 2025-10-10T02:02:56.8893069Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8893770Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8894476Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8895130Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8895490Z  2025-10-10T02:02:56.8895862Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8896548Z  test. 2025-10-10T02:02:56.8896885Z  """ 2025-10-10T02:02:56.8897535Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8898376Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8899023Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8899265Z 2025-10-10T02:02:56.8899422Z models/test_initialization.py:131: 2025-10-10T02:02:56.8899686Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8899982Z utils.py:892: in wrapper 2025-10-10T02:02:56.8900432Z raise original_exception 2025-10-10T02:02:56.8900907Z utils.py:836: in wrapper 2025-10-10T02:02:56.8901192Z func(*args, **kwargs) 2025-10-10T02:02:56.8901617Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8902075Z LLM( 2025-10-10T02:02:56.8902730Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8903288Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8903869Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8904457Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8905030Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8905591Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8906166Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8906764Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8907369Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8908249Z super().__init__( 2025-10-10T02:02:56.8908927Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8909698Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8910190Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8910362Z next(self.gen) 2025-10-10T02:02:56.8910708Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8910912Z wait_for_engine_startup( 2025-10-10T02:02:56.8911092Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8911099Z 2025-10-10T02:02:56.8911540Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8911931Z "See root cause above. " 2025-10-10T02:02:56.8912404Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8912865Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8912870Z 2025-10-10T02:02:56.8913263Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8913717Z ____________ test_can_initialize_large_subset[HCXVisionForCausalLM] ____________ 2025-10-10T02:02:56.8913723Z 2025-10-10T02:02:56.8913873Z model_arch = 'HCXVisionForCausalLM' 2025-10-10T02:02:56.8914088Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b1d00> 2025-10-10T02:02:56.8914091Z 2025-10-10T02:02:56.8914645Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8914993Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8915248Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8915489Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8915577Z  2025-10-10T02:02:56.8915906Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8916102Z  test. 2025-10-10T02:02:56.8916245Z  """ 2025-10-10T02:02:56.8916554Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8916906Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8917127Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8917133Z 2025-10-10T02:02:56.8917277Z models/test_initialization.py:131: 2025-10-10T02:02:56.8917450Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8917660Z utils.py:892: in wrapper 2025-10-10T02:02:56.8917839Z raise original_exception 2025-10-10T02:02:56.8917951Z utils.py:836: in wrapper 2025-10-10T02:02:56.8918084Z func(*args, **kwargs) 2025-10-10T02:02:56.8918263Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8918396Z LLM( 2025-10-10T02:02:56.8918866Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8919225Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8919601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8919826Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8920355Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8920696Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8921154Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8921514Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8921909Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8922269Z super().__init__( 2025-10-10T02:02:56.8922673Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8922932Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8923160Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8923330Z next(self.gen) 2025-10-10T02:02:56.8923752Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8923911Z wait_for_engine_startup( 2025-10-10T02:02:56.8924090Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8924096Z 2025-10-10T02:02:56.8924893Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8925219Z "See root cause above. " 2025-10-10T02:02:56.8925852Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8926275Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8926283Z 2025-10-10T02:02:56.8926602Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8926854Z _____________ test_can_initialize_large_subset[MistralForCausalLM] _____________ 2025-10-10T02:02:56.8926860Z 2025-10-10T02:02:56.8926945Z model_arch = 'MistralForCausalLM' 2025-10-10T02:02:56.8927123Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124f6cc0> 2025-10-10T02:02:56.8927129Z 2025-10-10T02:02:56.8931983Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8932328Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8932665Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8933071Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8933242Z  2025-10-10T02:02:56.8933523Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8933634Z  test. 2025-10-10T02:02:56.8933832Z  """ 2025-10-10T02:02:56.8934319Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8934797Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8935169Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8935175Z 2025-10-10T02:02:56.8935327Z models/test_initialization.py:131: 2025-10-10T02:02:56.8935432Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8935551Z utils.py:892: in wrapper 2025-10-10T02:02:56.8935722Z raise original_exception 2025-10-10T02:02:56.8935829Z utils.py:836: in wrapper 2025-10-10T02:02:56.8935964Z func(*args, **kwargs) 2025-10-10T02:02:56.8936273Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8936436Z LLM( 2025-10-10T02:02:56.8936843Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8937155Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8937676Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8937910Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8938353Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8939066Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8939428Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8940347Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8940837Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8941120Z super().__init__( 2025-10-10T02:02:56.8941468Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8942292Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8942569Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8943005Z next(self.gen) 2025-10-10T02:02:56.8943516Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8943950Z wait_for_engine_startup( 2025-10-10T02:02:56.8944077Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8944086Z 2025-10-10T02:02:56.8945850Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8946116Z "See root cause above. " 2025-10-10T02:02:56.8946487Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8946880Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8946885Z 2025-10-10T02:02:56.8947230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8947489Z _____________ test_can_initialize_large_subset[ArcticForCausalLM] ______________ 2025-10-10T02:02:56.8947496Z 2025-10-10T02:02:56.8947587Z model_arch = 'ArcticForCausalLM' 2025-10-10T02:02:56.8947765Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b01d0> 2025-10-10T02:02:56.8947768Z 2025-10-10T02:02:56.8952521Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8952883Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8953079Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8953312Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8953412Z  2025-10-10T02:02:56.8953678Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8953869Z  test. 2025-10-10T02:02:56.8954051Z  """ 2025-10-10T02:02:56.8954357Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8954705Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8954942Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8954991Z 2025-10-10T02:02:56.8955146Z models/test_initialization.py:131: 2025-10-10T02:02:56.8955255Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8955377Z utils.py:892: in wrapper 2025-10-10T02:02:56.8955543Z raise original_exception 2025-10-10T02:02:56.8955654Z utils.py:836: in wrapper 2025-10-10T02:02:56.8955792Z func(*args, **kwargs) 2025-10-10T02:02:56.8955976Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8956074Z LLM( 2025-10-10T02:02:56.8956445Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8957061Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8957439Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8957924Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8958248Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8959532Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8961526Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8962083Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8962673Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8963100Z super().__init__( 2025-10-10T02:02:56.8963778Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8964073Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8964310Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8964490Z next(self.gen) 2025-10-10T02:02:56.8964846Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8964996Z wait_for_engine_startup( 2025-10-10T02:02:56.8965162Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8965169Z 2025-10-10T02:02:56.8966406Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8966784Z "See root cause above. " 2025-10-10T02:02:56.8967173Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8967559Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8967564Z 2025-10-10T02:02:56.8968059Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8968392Z ____________ test_can_initialize_large_subset[StableLmForCausalLM] _____________ 2025-10-10T02:02:56.8968396Z 2025-10-10T02:02:56.8968482Z model_arch = 'StableLmForCausalLM' 2025-10-10T02:02:56.8968659Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249fd10> 2025-10-10T02:02:56.8968663Z 2025-10-10T02:02:56.8973060Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8973391Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8973577Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8973958Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8974071Z  2025-10-10T02:02:56.8974339Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8974511Z  test. 2025-10-10T02:02:56.8974633Z  """ 2025-10-10T02:02:56.8975022Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8975538Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8975966Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8975992Z 2025-10-10T02:02:56.8976163Z models/test_initialization.py:131: 2025-10-10T02:02:56.8976267Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8976398Z utils.py:892: in wrapper 2025-10-10T02:02:56.8976576Z raise original_exception 2025-10-10T02:02:56.8976683Z utils.py:836: in wrapper 2025-10-10T02:02:56.8976807Z func(*args, **kwargs) 2025-10-10T02:02:56.8977095Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8977295Z LLM( 2025-10-10T02:02:56.8977646Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.8977909Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.8978376Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.8979348Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.8979840Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.8980641Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.8981096Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.8982136Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.8982654Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.8983102Z super().__init__( 2025-10-10T02:02:56.8983600Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.8984443Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.8984926Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.8985780Z next(self.gen) 2025-10-10T02:02:56.8986210Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.8986355Z wait_for_engine_startup( 2025-10-10T02:02:56.8986453Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8986523Z 2025-10-10T02:02:56.8988857Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.8989206Z "See root cause above. " 2025-10-10T02:02:56.8989582Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.8989964Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.8989972Z 2025-10-10T02:02:56.8990376Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.8990653Z ________ test_can_initialize_large_subset[Emu3ForConditionalGeneration] ________ 2025-10-10T02:02:56.8990658Z 2025-10-10T02:02:56.8990777Z model_arch = 'Emu3ForConditionalGeneration' 2025-10-10T02:02:56.8990951Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b30e0> 2025-10-10T02:02:56.8990954Z 2025-10-10T02:02:56.8995454Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.8995788Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.8995980Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.8996405Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.8996565Z  2025-10-10T02:02:56.8996885Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.8996984Z  test. 2025-10-10T02:02:56.8997097Z  """ 2025-10-10T02:02:56.8997393Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.8997750Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.8998101Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.8998108Z 2025-10-10T02:02:56.8998354Z models/test_initialization.py:131: 2025-10-10T02:02:56.8998468Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.8998600Z utils.py:892: in wrapper 2025-10-10T02:02:56.8998810Z raise original_exception 2025-10-10T02:02:56.8999022Z utils.py:836: in wrapper 2025-10-10T02:02:56.8999304Z func(*args, **kwargs) 2025-10-10T02:02:56.8999501Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.8999761Z LLM( 2025-10-10T02:02:56.9000087Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9001689Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9002198Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9002719Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9003265Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9003933Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9004351Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9005238Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9005806Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9006007Z super().__init__( 2025-10-10T02:02:56.9006358Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9007195Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9007691Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9008023Z next(self.gen) 2025-10-10T02:02:56.9008548Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9008755Z wait_for_engine_startup( 2025-10-10T02:02:56.9008859Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9008863Z 2025-10-10T02:02:56.9011006Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9011351Z "See root cause above. " 2025-10-10T02:02:56.9011845Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9012244Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9012249Z 2025-10-10T02:02:56.9012714Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9012971Z _____________ test_can_initialize_large_subset[Exaone4ForCausalLM] _____________ 2025-10-10T02:02:56.9012976Z 2025-10-10T02:02:56.9013064Z model_arch = 'Exaone4ForCausalLM' 2025-10-10T02:02:56.9013235Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249de80> 2025-10-10T02:02:56.9013238Z 2025-10-10T02:02:56.9018022Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9018573Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9018770Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9018999Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9019107Z  2025-10-10T02:02:56.9019516Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9019627Z  test. 2025-10-10T02:02:56.9019748Z  """ 2025-10-10T02:02:56.9020043Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9020467Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9020875Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9020880Z 2025-10-10T02:02:56.9021042Z models/test_initialization.py:131: 2025-10-10T02:02:56.9021147Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9021259Z utils.py:892: in wrapper 2025-10-10T02:02:56.9021451Z raise original_exception 2025-10-10T02:02:56.9021688Z utils.py:836: in wrapper 2025-10-10T02:02:56.9021886Z func(*args, **kwargs) 2025-10-10T02:02:56.9022126Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9022228Z LLM( 2025-10-10T02:02:56.9022558Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9023551Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9024090Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9024656Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9025001Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9025834Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9026215Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9027615Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9028103Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9028534Z super().__init__( 2025-10-10T02:02:56.9028888Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9029947Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9030184Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9031056Z next(self.gen) 2025-10-10T02:02:56.9031419Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9031979Z wait_for_engine_startup( 2025-10-10T02:02:56.9032097Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9032103Z 2025-10-10T02:02:56.9034329Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9034605Z "See root cause above. " 2025-10-10T02:02:56.9034990Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9035498Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9035507Z 2025-10-10T02:02:56.9035870Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9036124Z ______________ test_can_initialize_large_subset[Glm4ForCausalLM] _______________ 2025-10-10T02:02:56.9036128Z 2025-10-10T02:02:56.9036215Z model_arch = 'Glm4ForCausalLM' 2025-10-10T02:02:56.9036478Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b0920> 2025-10-10T02:02:56.9036523Z 2025-10-10T02:02:56.9041781Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9042606Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9043193Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9043988Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9044409Z  2025-10-10T02:02:56.9044902Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9045281Z  test. 2025-10-10T02:02:56.9045617Z  """ 2025-10-10T02:02:56.9046140Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9046913Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9047720Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9047968Z 2025-10-10T02:02:56.9048113Z models/test_initialization.py:131: 2025-10-10T02:02:56.9048500Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9048871Z utils.py:892: in wrapper 2025-10-10T02:02:56.9049235Z raise original_exception 2025-10-10T02:02:56.9049534Z utils.py:836: in wrapper 2025-10-10T02:02:56.9049923Z func(*args, **kwargs) 2025-10-10T02:02:56.9050365Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9050670Z LLM( 2025-10-10T02:02:56.9051241Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9051790Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9052387Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9053070Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9053759Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9054307Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9061685Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9062372Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9063035Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9063936Z super().__init__( 2025-10-10T02:02:56.9064778Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9065524Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9066015Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9066426Z next(self.gen) 2025-10-10T02:02:56.9067181Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9067744Z wait_for_engine_startup( 2025-10-10T02:02:56.9068149Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9068414Z 2025-10-10T02:02:56.9069102Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9069748Z "See root cause above. " 2025-10-10T02:02:56.9070491Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9071277Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9071805Z 2025-10-10T02:02:56.9072142Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9072725Z ___________ test_can_initialize_large_subset[BailingMoeForCausalLM] ____________ 2025-10-10T02:02:56.9073126Z 2025-10-10T02:02:56.9073228Z model_arch = 'BailingMoeForCausalLM' 2025-10-10T02:02:56.9073615Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221a956030> 2025-10-10T02:02:56.9073967Z 2025-10-10T02:02:56.9074406Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9075211Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9075721Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9076169Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9076619Z  2025-10-10T02:02:56.9076985Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9077512Z  test. 2025-10-10T02:02:56.9077756Z  """ 2025-10-10T02:02:56.9078195Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9078817Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9079465Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9079848Z 2025-10-10T02:02:56.9080079Z models/test_initialization.py:131: 2025-10-10T02:02:56.9080354Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9080649Z utils.py:892: in wrapper 2025-10-10T02:02:56.9080951Z raise original_exception 2025-10-10T02:02:56.9081253Z utils.py:836: in wrapper 2025-10-10T02:02:56.9081520Z func(*args, **kwargs) 2025-10-10T02:02:56.9081857Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9082153Z LLM( 2025-10-10T02:02:56.9082608Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9083131Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9083721Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9084308Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9084863Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9085416Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9086061Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9086716Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9087310Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9087831Z super().__init__( 2025-10-10T02:02:56.9088411Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9088971Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9089440Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9089849Z next(self.gen) 2025-10-10T02:02:56.9090381Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9090868Z wait_for_engine_startup( 2025-10-10T02:02:56.9091167Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9091345Z 2025-10-10T02:02:56.9091752Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9092373Z "See root cause above. " 2025-10-10T02:02:56.9092956Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9093645Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9094017Z 2025-10-10T02:02:56.9094342Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9094900Z ______________ test_can_initialize_large_subset[Gemma3TextModel] _______________ 2025-10-10T02:02:56.9095152Z 2025-10-10T02:02:56.9095232Z model_arch = 'Gemma3TextModel' 2025-10-10T02:02:56.9095539Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b0740> 2025-10-10T02:02:56.9095781Z 2025-10-10T02:02:56.9096358Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9097025Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9097506Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9097924Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9098251Z  2025-10-10T02:02:56.9098607Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9098970Z  test. 2025-10-10T02:02:56.9099201Z  """ 2025-10-10T02:02:56.9099624Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9100262Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9100801Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9101045Z 2025-10-10T02:02:56.9101178Z models/test_initialization.py:131: 2025-10-10T02:02:56.9101443Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9101824Z utils.py:892: in wrapper 2025-10-10T02:02:56.9102211Z raise original_exception 2025-10-10T02:02:56.9102501Z utils.py:836: in wrapper 2025-10-10T02:02:56.9102756Z func(*args, **kwargs) 2025-10-10T02:02:56.9103083Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9103380Z LLM( 2025-10-10T02:02:56.9103812Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9104419Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9105004Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9105569Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9106117Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9106657Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9107299Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9107901Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9108491Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9108987Z super().__init__( 2025-10-10T02:02:56.9109494Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9110061Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9110544Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9110953Z next(self.gen) 2025-10-10T02:02:56.9111497Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9111970Z wait_for_engine_startup( 2025-10-10T02:02:56.9112223Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9112399Z 2025-10-10T02:02:56.9112800Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9113401Z "See root cause above. " 2025-10-10T02:02:56.9113990Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9114684Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9115053Z 2025-10-10T02:02:56.9115368Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9115923Z _____________ test_can_initialize_large_subset[MiniCPMForCausalLM] _____________ 2025-10-10T02:02:56.9116201Z 2025-10-10T02:02:56.9116288Z model_arch = 'MiniCPMForCausalLM' 2025-10-10T02:02:56.9116600Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249f680> 2025-10-10T02:02:56.9116840Z 2025-10-10T02:02:56.9117212Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9117911Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9118448Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9118877Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9119325Z  2025-10-10T02:02:56.9119680Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9120045Z  test. 2025-10-10T02:02:56.9120415Z  """ 2025-10-10T02:02:56.9120846Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9121482Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9122024Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9122267Z 2025-10-10T02:02:56.9122402Z models/test_initialization.py:131: 2025-10-10T02:02:56.9122677Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9123008Z utils.py:892: in wrapper 2025-10-10T02:02:56.9123310Z raise original_exception 2025-10-10T02:02:56.9123602Z utils.py:836: in wrapper 2025-10-10T02:02:56.9123853Z func(*args, **kwargs) 2025-10-10T02:02:56.9124191Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9124491Z LLM( 2025-10-10T02:02:56.9124923Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9125448Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9126030Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9126617Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9127183Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9127729Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9128300Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9128899Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9129494Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9130004Z super().__init__( 2025-10-10T02:02:56.9130515Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9131075Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9131545Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9131954Z next(self.gen) 2025-10-10T02:02:56.9132480Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9132959Z wait_for_engine_startup( 2025-10-10T02:02:56.9133212Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9133389Z 2025-10-10T02:02:56.9133793Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9134446Z "See root cause above. " 2025-10-10T02:02:56.9135082Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9135784Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9136155Z 2025-10-10T02:02:56.9136471Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9137086Z ____________ test_can_initialize_large_subset[NemotronHForCausalLM] ____________ 2025-10-10T02:02:56.9137344Z 2025-10-10T02:02:56.9137438Z model_arch = 'NemotronHForCausalLM' 2025-10-10T02:02:56.9137757Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124b18e0> 2025-10-10T02:02:56.9137997Z 2025-10-10T02:02:56.9138672Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9139598Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9140106Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9140716Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9141062Z  2025-10-10T02:02:56.9141589Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9141972Z  test. 2025-10-10T02:02:56.9142203Z  """ 2025-10-10T02:02:56.9142637Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9143443Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9144004Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9144394Z 2025-10-10T02:02:56.9144572Z models/test_initialization.py:131: 2025-10-10T02:02:56.9144832Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9145128Z utils.py:892: in wrapper 2025-10-10T02:02:56.9145441Z raise original_exception 2025-10-10T02:02:56.9145797Z utils.py:836: in wrapper 2025-10-10T02:02:56.9146149Z func(*args, **kwargs) 2025-10-10T02:02:56.9146468Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9146766Z LLM( 2025-10-10T02:02:56.9147207Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9147945Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9148542Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9149271Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9149827Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9150540Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9151115Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9151860Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9152610Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9153291Z super().__init__( 2025-10-10T02:02:56.9153815Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9154422Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9155142Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9155632Z next(self.gen) 2025-10-10T02:02:56.9156481Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9157003Z wait_for_engine_startup( 2025-10-10T02:02:56.9157260Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9157431Z 2025-10-10T02:02:56.9157843Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9158508Z "See root cause above. " 2025-10-10T02:02:56.9159210Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9159909Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9160279Z 2025-10-10T02:02:56.9160593Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9161148Z __________ test_can_initialize_large_subset[Ernie4_5_MoeForCausalLM] ___________ 2025-10-10T02:02:56.9161401Z 2025-10-10T02:02:56.9161501Z model_arch = 'Ernie4_5_MoeForCausalLM' 2025-10-10T02:02:56.9161911Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63cfb0> 2025-10-10T02:02:56.9162372Z 2025-10-10T02:02:56.9163432Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9164291Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9164789Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9165272Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9165761Z  2025-10-10T02:02:56.9166204Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9166662Z  test. 2025-10-10T02:02:56.9166892Z  """ 2025-10-10T02:02:56.9167334Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9168122Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9168675Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9168919Z 2025-10-10T02:02:56.9169056Z models/test_initialization.py:131: 2025-10-10T02:02:56.9169456Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9169751Z utils.py:892: in wrapper 2025-10-10T02:02:56.9170049Z raise original_exception 2025-10-10T02:02:56.9170335Z utils.py:836: in wrapper 2025-10-10T02:02:56.9170587Z func(*args, **kwargs) 2025-10-10T02:02:56.9171061Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9171441Z LLM( 2025-10-10T02:02:56.9171934Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9172495Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9173237Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9173894Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9174609Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9175162Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9175910Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9176558Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9177358Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9178072Z super().__init__( 2025-10-10T02:02:56.9178591Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9179146Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9179821Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9180388Z next(self.gen) 2025-10-10T02:02:56.9180970Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9181647Z wait_for_engine_startup( 2025-10-10T02:02:56.9181908Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9182081Z 2025-10-10T02:02:56.9182494Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9183091Z "See root cause above. " 2025-10-10T02:02:56.9183664Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9184351Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9184724Z 2025-10-10T02:02:56.9185044Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9185592Z ______________ test_can_initialize_large_subset[MambaForCausalLM] ______________ 2025-10-10T02:02:56.9185841Z 2025-10-10T02:02:56.9185922Z model_arch = 'MambaForCausalLM' 2025-10-10T02:02:56.9186233Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63c110> 2025-10-10T02:02:56.9186475Z 2025-10-10T02:02:56.9188368Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9189311Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9189816Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9190317Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9190793Z  2025-10-10T02:02:56.9191418Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9191872Z  test. 2025-10-10T02:02:56.9192103Z  """ 2025-10-10T02:02:56.9192551Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9193411Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9194191Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9194528Z 2025-10-10T02:02:56.9194681Z models/test_initialization.py:131: 2025-10-10T02:02:56.9194957Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9195236Z utils.py:892: in wrapper 2025-10-10T02:02:56.9195597Z raise original_exception 2025-10-10T02:02:56.9196257Z utils.py:836: in wrapper 2025-10-10T02:02:56.9196550Z func(*args, **kwargs) 2025-10-10T02:02:56.9196991Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9197298Z LLM( 2025-10-10T02:02:56.9197980Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9198520Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9199329Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9199912Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9200567Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9201149Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9201722Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9202504Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9203212Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9203710Z super().__init__( 2025-10-10T02:02:56.9204217Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9204981Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9205483Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9205888Z next(self.gen) 2025-10-10T02:02:56.9206681Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9207176Z wait_for_engine_startup( 2025-10-10T02:02:56.9207440Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9207615Z 2025-10-10T02:02:56.9208020Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9208631Z "See root cause above. " 2025-10-10T02:02:56.9209206Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9209998Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9210451Z 2025-10-10T02:02:56.9210784Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9211383Z ___ test_can_initialize_large_subset[LlavaNextVideoForConditionalGeneration] ___ 2025-10-10T02:02:56.9211674Z 2025-10-10T02:02:56.9211797Z model_arch = 'LlavaNextVideoForConditionalGeneration' 2025-10-10T02:02:56.9212415Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249e990> 2025-10-10T02:02:56.9212988Z 2025-10-10T02:02:56.9213475Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9214259Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9214751Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9215169Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9215571Z  2025-10-10T02:02:56.9216057Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9216498Z  test. 2025-10-10T02:02:56.9216772Z  """ 2025-10-10T02:02:56.9217196Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9217817Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9218511Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9218748Z 2025-10-10T02:02:56.9218886Z models/test_initialization.py:131: 2025-10-10T02:02:56.9219145Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9219579Z utils.py:892: in wrapper 2025-10-10T02:02:56.9219894Z raise original_exception 2025-10-10T02:02:56.9220182Z utils.py:836: in wrapper 2025-10-10T02:02:56.9220428Z func(*args, **kwargs) 2025-10-10T02:02:56.9220753Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9221139Z LLM( 2025-10-10T02:02:56.9221610Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9222134Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9222847Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9223432Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9224022Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:56.9224678Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:56.9225238Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:56.9225923Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:56.9226551Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:56.9227203Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:56.9227966Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:56.9228690Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:56.9229570Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:56.9230292Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:56.9231127Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:56.9231795Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:56.9232503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:56.9233046Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:56.9233673Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:56.9234424Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:56.9235136Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:56.9235919Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:56.9236541Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:56.9237289Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:56.9237668Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9237845Z 2025-10-10T02:02:56.9238035Z > raise ValueError( 2025-10-10T02:02:56.9238777Z f"Model architectures {architectures} failed " 2025-10-10T02:02:56.9239611Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:56.9240399Z E ValueError: Model architectures ['LlavaNextVideoForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:56.9240866Z 2025-10-10T02:02:56.9241227Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:56.9241820Z ______________ test_can_initialize_large_subset[GemmaForCausalLM] ______________ 2025-10-10T02:02:56.9242071Z 2025-10-10T02:02:56.9242150Z model_arch = 'GemmaForCausalLM' 2025-10-10T02:02:56.9242446Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa62bbf0> 2025-10-10T02:02:56.9242682Z 2025-10-10T02:02:56.9246234Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9247035Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9247525Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9247944Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9248393Z  2025-10-10T02:02:56.9248767Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9249228Z  test. 2025-10-10T02:02:56.9249485Z  """ 2025-10-10T02:02:56.9249991Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9250698Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9251341Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9251580Z 2025-10-10T02:02:56.9251721Z models/test_initialization.py:131: 2025-10-10T02:02:56.9252065Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9252461Z utils.py:892: in wrapper 2025-10-10T02:02:56.9252761Z raise original_exception 2025-10-10T02:02:56.9253048Z utils.py:836: in wrapper 2025-10-10T02:02:56.9253324Z func(*args, **kwargs) 2025-10-10T02:02:56.9253841Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9254221Z LLM( 2025-10-10T02:02:56.9254737Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9255374Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9256095Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9256682Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9257359Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9257907Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9258621Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9259240Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9259980Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9260623Z super().__init__( 2025-10-10T02:02:56.9261146Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9261724Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9262194Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9262735Z next(self.gen) 2025-10-10T02:02:56.9263384Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9263990Z wait_for_engine_startup( 2025-10-10T02:02:56.9264373Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9264555Z 2025-10-10T02:02:56.9264972Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9265591Z "See root cause above. " 2025-10-10T02:02:56.9266170Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9266866Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9267236Z 2025-10-10T02:02:56.9267558Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9268186Z ________________ test_can_initialize_large_subset[Gemma2Model] _________________ 2025-10-10T02:02:56.9268472Z 2025-10-10T02:02:56.9268552Z model_arch = 'Gemma2Model' 2025-10-10T02:02:56.9268843Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221249e5a0> 2025-10-10T02:02:56.9269083Z 2025-10-10T02:02:56.9271244Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9272305Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9273305Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9274121Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9274467Z  2025-10-10T02:02:56.9274838Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9275398Z  test. 2025-10-10T02:02:56.9275643Z  """ 2025-10-10T02:02:56.9276166Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9276954Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9277515Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9277837Z 2025-10-10T02:02:56.9278069Z models/test_initialization.py:131: 2025-10-10T02:02:56.9278351Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9278653Z utils.py:892: in wrapper 2025-10-10T02:02:56.9278964Z raise original_exception 2025-10-10T02:02:56.9279378Z utils.py:836: in wrapper 2025-10-10T02:02:56.9279781Z func(*args, **kwargs) 2025-10-10T02:02:56.9280169Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9280476Z LLM( 2025-10-10T02:02:56.9281066Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9281614Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9282247Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9282945Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9283597Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9284221Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9285081Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9285708Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9286306Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9287106Z super().__init__( 2025-10-10T02:02:56.9287632Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9288474Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9288958Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9289378Z next(self.gen) 2025-10-10T02:02:56.9289979Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9290500Z wait_for_engine_startup( 2025-10-10T02:02:56.9290769Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9290945Z 2025-10-10T02:02:56.9291346Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9291949Z "See root cause above. " 2025-10-10T02:02:56.9292575Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9293277Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9293891Z 2025-10-10T02:02:56.9294489Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9295210Z ______________ test_can_initialize_large_subset[Qwen2_5OmniModel] ______________ 2025-10-10T02:02:56.9295598Z 2025-10-10T02:02:56.9295680Z model_arch = 'Qwen2_5OmniModel' 2025-10-10T02:02:56.9295982Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa62b080> 2025-10-10T02:02:56.9296418Z 2025-10-10T02:02:56.9296838Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9297673Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9298306Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9298734Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9299063Z  2025-10-10T02:02:56.9299424Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9299945Z  test. 2025-10-10T02:02:56.9300177Z  """ 2025-10-10T02:02:56.9300605Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9301387Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9301946Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9302186Z 2025-10-10T02:02:56.9302321Z models/test_initialization.py:131: 2025-10-10T02:02:56.9302727Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9303032Z utils.py:892: in wrapper 2025-10-10T02:02:56.9303353Z raise original_exception 2025-10-10T02:02:56.9303654Z utils.py:836: in wrapper 2025-10-10T02:02:56.9303908Z func(*args, **kwargs) 2025-10-10T02:02:56.9304375Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9304699Z LLM( 2025-10-10T02:02:56.9305139Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9305816Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9306404Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9307067Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9307802Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9308677Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9309448Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9310167Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9310763Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9311643Z super().__init__( 2025-10-10T02:02:56.9312354Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9313376Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9313943Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9314377Z next(self.gen) 2025-10-10T02:02:56.9315026Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9315524Z wait_for_engine_startup( 2025-10-10T02:02:56.9315784Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9315966Z 2025-10-10T02:02:56.9316375Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9316981Z "See root cause above. " 2025-10-10T02:02:56.9317562Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9318258Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9318652Z 2025-10-10T02:02:56.9319332Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9320225Z ________ test_can_initialize_large_subset[GraniteMoeHybridForCausalLM] _________ 2025-10-10T02:02:56.9320509Z 2025-10-10T02:02:56.9320636Z model_arch = 'GraniteMoeHybridForCausalLM' 2025-10-10T02:02:56.9321164Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63c0b0> 2025-10-10T02:02:56.9321409Z 2025-10-10T02:02:56.9321807Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9322759Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9323498Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9323984Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9324307Z  2025-10-10T02:02:56.9324670Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9325274Z  test. 2025-10-10T02:02:56.9325517Z  """ 2025-10-10T02:02:56.9325965Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9326827Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9327057Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9327062Z 2025-10-10T02:02:56.9327281Z models/test_initialization.py:131: 2025-10-10T02:02:56.9327420Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9327545Z utils.py:892: in wrapper 2025-10-10T02:02:56.9327784Z raise original_exception 2025-10-10T02:02:56.9328000Z utils.py:836: in wrapper 2025-10-10T02:02:56.9328192Z func(*args, **kwargs) 2025-10-10T02:02:56.9328382Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9328557Z LLM( 2025-10-10T02:02:56.9328887Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9329108Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9329538Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9329930Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9330344Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9330577Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9331068Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9331481Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9331833Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9332015Z super().__init__( 2025-10-10T02:02:56.9332439Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9333419Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9333749Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9334841Z next(self.gen) 2025-10-10T02:02:56.9335258Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9335630Z wait_for_engine_startup( 2025-10-10T02:02:56.9335746Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9335752Z 2025-10-10T02:02:56.9338079Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9338343Z "See root cause above. " 2025-10-10T02:02:56.9338713Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9339244Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9339250Z 2025-10-10T02:02:56.9339596Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9339863Z ___________ test_can_initialize_large_subset[Llama_Nemotron_Nano_VL] ___________ 2025-10-10T02:02:56.9339867Z 2025-10-10T02:02:56.9339952Z model_arch = 'Llama_Nemotron_Nano_VL' 2025-10-10T02:02:56.9340133Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa622a50> 2025-10-10T02:02:56.9340137Z 2025-10-10T02:02:56.9345403Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9346039Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9346281Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9346580Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9346691Z  2025-10-10T02:02:56.9347095Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9347224Z  test. 2025-10-10T02:02:56.9347362Z  """ 2025-10-10T02:02:56.9347736Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9348303Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9348632Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9348644Z 2025-10-10T02:02:56.9348920Z models/test_initialization.py:131: 2025-10-10T02:02:56.9349143Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9349308Z utils.py:892: in wrapper 2025-10-10T02:02:56.9349515Z raise original_exception 2025-10-10T02:02:56.9349656Z utils.py:836: in wrapper 2025-10-10T02:02:56.9349819Z func(*args, **kwargs) 2025-10-10T02:02:56.9350040Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9350166Z LLM( 2025-10-10T02:02:56.9350619Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9351009Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9351474Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:169: in from_engine_args 2025-10-10T02:02:56.9352494Z vllm_config = engine_args.create_engine_config(usage_context) 2025-10-10T02:02:56.9352978Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/engine/arg_utils.py:1134: in create_engine_config 2025-10-10T02:02:56.9353887Z model_config = self.create_model_config() 2025-10-10T02:02:56.9354304Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/engine/arg_utils.py:994: in create_model_config 2025-10-10T02:02:56.9354903Z return ModelConfig( 2025-10-10T02:02:56.9355348Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py:123: in __init__ 2025-10-10T02:02:56.9357779Z s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s) 2025-10-10T02:02:56.9358137Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/config/model.py:437: in __post_init__ 2025-10-10T02:02:56.9359963Z hf_config = get_config(self.hf_config_path or self.model, 2025-10-10T02:02:56.9360344Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/transformers_utils/config.py:584: in get_config 2025-10-10T02:02:56.9361355Z config_dict, config = config_parser.parse( 2025-10-10T02:02:56.9363312Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/transformers_utils/config.py:127: in parse 2025-10-10T02:02:56.9363500Z config = config_class.from_pretrained( 2025-10-10T02:02:56.9363886Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/transformers/configuration_utils.py:640: in from_pretrained 2025-10-10T02:02:56.9365268Z return cls.from_dict(config_dict, **kwargs) 2025-10-10T02:02:56.9365700Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/transformers/configuration_utils.py:808: in from_dict 2025-10-10T02:02:56.9366563Z config = cls(**config_dict) 2025-10-10T02:02:56.9367062Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/transformers_utils/configs/nemotron_vl.py:39: in __init__ 2025-10-10T02:02:56.9370401Z vision_auto_config = get_class_from_dynamic_module(*vision_config["auto_map"]["AutoConfig"].split("--")[::-1]) 2025-10-10T02:02:56.9370878Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/transformers/dynamic_module_utils.py:581: in get_class_from_dynamic_module 2025-10-10T02:02:56.9371709Z return get_class_in_module(class_name, final_module, force_reload=force_download) 2025-10-10T02:02:56.9372213Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/transformers/dynamic_module_utils.py:276: in get_class_in_module 2025-10-10T02:02:56.9372988Z module_spec.loader.exec_module(module) 2025-10-10T02:02:56.9373242Z :999: in exec_module 2025-10-10T02:02:56.9374225Z ??? 2025-10-10T02:02:56.9374475Z :488: in _call_with_frames_removed 2025-10-10T02:02:56.9375688Z ??? 2025-10-10T02:02:56.9376235Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/hf_model.py:23: in 2025-10-10T02:02:56.9377189Z from .common import RESOURCE_MAP, DEFAULT_VERSION 2025-10-10T02:02:56.9377701Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/common.py:12: in 2025-10-10T02:02:56.9378159Z from .radio_model import Resolution 2025-10-10T02:02:56.9378700Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/radio_model.py:15: in 2025-10-10T02:02:56.9379387Z from .enable_cpe_support import enable_cpe 2025-10-10T02:02:56.9379950Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/enable_cpe_support.py:22: in 2025-10-10T02:02:56.9380616Z from .dual_hybrid_vit import HybridModel 2025-10-10T02:02:56.9381152Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/dual_hybrid_vit.py:14: in 2025-10-10T02:02:56.9381999Z from . import extra_timm_models as et 2025-10-10T02:02:56.9382535Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/extra_timm_models.py:26: in 2025-10-10T02:02:56.9383006Z from . import dinov2_arch 2025-10-10T02:02:56.9383597Z ../../../.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-H/8136123fd0f00a34e597f619f2c0ee2059de0714/dinov2_arch.py:35: in 2025-10-10T02:02:56.9385847Z from xformers.ops import fmha, scaled_index_add, index_select_cat, SwiGLU, memory_efficient_attention, unbind 2025-10-10T02:02:56.9386295Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/__init__.py:9: in 2025-10-10T02:02:56.9386570Z from .fmha import ( 2025-10-10T02:02:56.9386975Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/__init__.py:10: in 2025-10-10T02:02:56.9388967Z from . import attn_bias, ck, ck_splitk, cutlass, flash, flash3, triton_splitk 2025-10-10T02:02:56.9389353Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/ops/fmha/triton_splitk.py:124: in 2025-10-10T02:02:56.9390164Z if TYPE_CHECKING or _is_triton_available(): 2025-10-10T02:02:56.9390564Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py:38: in func_wrapper 2025-10-10T02:02:56.9391412Z value = func() 2025-10-10T02:02:56.9391866Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/xformers/__init__.py:54: in _is_triton_available 2025-10-10T02:02:56.9394330Z if torch.cuda.get_device_capability("cuda") < (8, 0): 2025-10-10T02:02:56.9394713Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:602: in get_device_capability 2025-10-10T02:02:56.9395408Z prop = get_device_properties(device) 2025-10-10T02:02:56.9395840Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:619: in get_device_properties 2025-10-10T02:02:56.9396897Z _lazy_init() # will define _get_device_properties 2025-10-10T02:02:56.9397018Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9397024Z 2025-10-10T02:02:56.9398885Z > raise RuntimeError( 2025-10-10T02:02:56.9399477Z "Cannot re-initialize CUDA in forked subprocess. To use CUDA with " 2025-10-10T02:02:56.9400034Z "multiprocessing, you must use the 'spawn' start method" 2025-10-10T02:02:56.9400128Z ) 2025-10-10T02:02:56.9400680Z E RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:56.9400688Z 2025-10-10T02:02:56.9401017Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:399: RuntimeError 2025-10-10T02:02:56.9401267Z _____________ test_can_initialize_large_subset[XverseForCausalLM] ______________ 2025-10-10T02:02:56.9401270Z 2025-10-10T02:02:56.9401355Z model_arch = 'XverseForCausalLM' 2025-10-10T02:02:56.9401531Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63c110> 2025-10-10T02:02:56.9401537Z 2025-10-10T02:02:56.9406746Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9407180Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9407367Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9407707Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9407871Z  2025-10-10T02:02:56.9408159Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9408343Z  test. 2025-10-10T02:02:56.9408459Z  """ 2025-10-10T02:02:56.9408749Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9409183Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9409404Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9409409Z 2025-10-10T02:02:56.9409549Z models/test_initialization.py:131: 2025-10-10T02:02:56.9409650Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9409770Z utils.py:892: in wrapper 2025-10-10T02:02:56.9410008Z raise original_exception 2025-10-10T02:02:56.9410119Z utils.py:836: in wrapper 2025-10-10T02:02:56.9410240Z func(*args, **kwargs) 2025-10-10T02:02:56.9410414Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9410620Z LLM( 2025-10-10T02:02:56.9410943Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9412584Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9412954Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9413825Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9414182Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9415506Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9415867Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9417294Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9417647Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9418447Z super().__init__( 2025-10-10T02:02:56.9418776Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9420187Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9420498Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9421377Z next(self.gen) 2025-10-10T02:02:56.9421742Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9421951Z wait_for_engine_startup( 2025-10-10T02:02:56.9422047Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9422062Z 2025-10-10T02:02:56.9424616Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9424995Z "See root cause above. " 2025-10-10T02:02:56.9425560Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9426127Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9426133Z 2025-10-10T02:02:56.9426477Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9426751Z ______ test_can_initialize_large_subset[InternS1ForConditionalGeneration] ______ 2025-10-10T02:02:56.9426810Z 2025-10-10T02:02:56.9426926Z model_arch = 'InternS1ForConditionalGeneration' 2025-10-10T02:02:56.9427103Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa62a030> 2025-10-10T02:02:56.9427107Z 2025-10-10T02:02:56.9432326Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9432878Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9433259Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9433650Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9433807Z  2025-10-10T02:02:56.9434247Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9434419Z  test. 2025-10-10T02:02:56.9434605Z  """ 2025-10-10T02:02:56.9434922Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9435529Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9435921Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9435951Z 2025-10-10T02:02:56.9436184Z models/test_initialization.py:131: 2025-10-10T02:02:56.9436351Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9436542Z utils.py:892: in wrapper 2025-10-10T02:02:56.9436815Z raise original_exception 2025-10-10T02:02:56.9436981Z utils.py:836: in wrapper 2025-10-10T02:02:56.9437107Z func(*args, **kwargs) 2025-10-10T02:02:56.9437293Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9437391Z LLM( 2025-10-10T02:02:56.9437905Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9438284Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9438875Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9439432Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9440014Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9440999Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9441605Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9443052Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9443645Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9443949Z super().__init__( 2025-10-10T02:02:56.9444607Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9445483Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9445889Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9446611Z next(self.gen) 2025-10-10T02:02:56.9447224Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9447537Z wait_for_engine_startup( 2025-10-10T02:02:56.9447702Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9447709Z 2025-10-10T02:02:56.9450190Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9450599Z "See root cause above. " 2025-10-10T02:02:56.9451273Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9451948Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9451959Z 2025-10-10T02:02:56.9452312Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9452587Z _____ test_can_initialize_large_subset[PaliGemmaForConditionalGeneration] ______ 2025-10-10T02:02:56.9452591Z 2025-10-10T02:02:56.9452706Z model_arch = 'PaliGemmaForConditionalGeneration' 2025-10-10T02:02:56.9452880Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63eff0> 2025-10-10T02:02:56.9452883Z 2025-10-10T02:02:56.9457513Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9458065Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9458375Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9458796Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9458951Z  2025-10-10T02:02:56.9459392Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9459563Z  test. 2025-10-10T02:02:56.9459700Z  """ 2025-10-10T02:02:56.9460088Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9460690Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9461084Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9461090Z 2025-10-10T02:02:56.9461329Z models/test_initialization.py:131: 2025-10-10T02:02:56.9461493Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9461689Z utils.py:892: in wrapper 2025-10-10T02:02:56.9461899Z raise original_exception 2025-10-10T02:02:56.9462013Z utils.py:836: in wrapper 2025-10-10T02:02:56.9462133Z func(*args, **kwargs) 2025-10-10T02:02:56.9462312Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9462416Z LLM( 2025-10-10T02:02:56.9463076Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9463506Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9464126Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9464529Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9465089Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9465897Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9466500Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9467571Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9468157Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9468716Z super().__init__( 2025-10-10T02:02:56.9469367Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9470338Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9470747Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9471473Z next(self.gen) 2025-10-10T02:02:56.9472090Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9472330Z wait_for_engine_startup( 2025-10-10T02:02:56.9472490Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9472501Z 2025-10-10T02:02:56.9474974Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9475379Z "See root cause above. " 2025-10-10T02:02:56.9475998Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9476582Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9476587Z 2025-10-10T02:02:56.9476914Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9477172Z _________ test_can_initialize_large_subset[RForConditionalGeneration] __________ 2025-10-10T02:02:56.9477179Z 2025-10-10T02:02:56.9477273Z model_arch = 'RForConditionalGeneration' 2025-10-10T02:02:56.9477446Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa674f20> 2025-10-10T02:02:56.9477451Z 2025-10-10T02:02:56.9482274Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9482814Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9483084Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9483417Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9483534Z  2025-10-10T02:02:56.9483805Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9483901Z  test. 2025-10-10T02:02:56.9484090Z  """ 2025-10-10T02:02:56.9484437Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9484773Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9485052Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9485123Z 2025-10-10T02:02:56.9485321Z models/test_initialization.py:131: 2025-10-10T02:02:56.9485425Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9485536Z utils.py:892: in wrapper 2025-10-10T02:02:56.9485826Z raise original_exception 2025-10-10T02:02:56.9485943Z utils.py:836: in wrapper 2025-10-10T02:02:56.9486059Z func(*args, **kwargs) 2025-10-10T02:02:56.9486245Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9486345Z LLM( 2025-10-10T02:02:56.9486718Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9487637Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9488051Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9489024Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9489399Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:56.9490917Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:56.9491271Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:56.9492252Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:56.9492672Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:56.9494071Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:56.9494480Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:56.9495543Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:56.9495962Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:56.9497694Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:56.9498126Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:56.9499353Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:56.9499739Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:56.9500956Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:56.9501558Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:56.9502240Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:56.9502678Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:56.9504262Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:56.9504869Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:56.9505426Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:56.9505530Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9505635Z 2025-10-10T02:02:56.9507418Z > raise ValueError( 2025-10-10T02:02:56.9508176Z f"Model architectures {architectures} failed " 2025-10-10T02:02:56.9508536Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:56.9508989Z E ValueError: Model architectures ['RForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:56.9508996Z 2025-10-10T02:02:56.9509468Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:56.9509720Z _____________ test_can_initialize_large_subset[DeciLMForCausalLM] ______________ 2025-10-10T02:02:56.9509727Z 2025-10-10T02:02:56.9509810Z model_arch = 'DeciLMForCausalLM' 2025-10-10T02:02:56.9509998Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa622e40> 2025-10-10T02:02:56.9510001Z 2025-10-10T02:02:56.9515013Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9515459Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9515667Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9515907Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9515999Z  2025-10-10T02:02:56.9516257Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9516355Z  test. 2025-10-10T02:02:56.9516469Z  """ 2025-10-10T02:02:56.9516894Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9517355Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9517601Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9517607Z 2025-10-10T02:02:56.9517853Z models/test_initialization.py:131: 2025-10-10T02:02:56.9517957Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9518074Z utils.py:892: in wrapper 2025-10-10T02:02:56.9518258Z raise original_exception 2025-10-10T02:02:56.9518365Z utils.py:836: in wrapper 2025-10-10T02:02:56.9518485Z func(*args, **kwargs) 2025-10-10T02:02:56.9518665Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9518764Z LLM( 2025-10-10T02:02:56.9519197Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9520718Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9521230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9521747Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9522224Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9523272Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9523631Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9524920Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9525453Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9525858Z super().__init__( 2025-10-10T02:02:56.9526198Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9527609Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9528091Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9528698Z next(self.gen) 2025-10-10T02:02:56.9529173Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9529322Z wait_for_engine_startup( 2025-10-10T02:02:56.9529426Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9529431Z 2025-10-10T02:02:56.9531852Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9532291Z "See root cause above. " 2025-10-10T02:02:56.9532852Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9533252Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9533257Z 2025-10-10T02:02:56.9533585Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9533847Z _____________ test_can_initialize_large_subset[Gemma3ForCausalLM] ______________ 2025-10-10T02:02:56.9533851Z 2025-10-10T02:02:56.9533934Z model_arch = 'Gemma3ForCausalLM' 2025-10-10T02:02:56.9534107Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63ac00> 2025-10-10T02:02:56.9534111Z 2025-10-10T02:02:56.9539196Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9539527Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9539769Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9540075Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9540172Z  2025-10-10T02:02:56.9540436Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9540531Z  test. 2025-10-10T02:02:56.9540644Z  """ 2025-10-10T02:02:56.9540935Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9541501Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9541831Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9541836Z 2025-10-10T02:02:56.9541979Z models/test_initialization.py:131: 2025-10-10T02:02:56.9542129Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9542339Z utils.py:892: in wrapper 2025-10-10T02:02:56.9542601Z raise original_exception 2025-10-10T02:02:56.9542782Z utils.py:836: in wrapper 2025-10-10T02:02:56.9542907Z func(*args, **kwargs) 2025-10-10T02:02:56.9543091Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9543188Z LLM( 2025-10-10T02:02:56.9543512Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9544867Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9545537Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9546005Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9546351Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9547728Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9548080Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9549411Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9549963Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9550418Z super().__init__( 2025-10-10T02:02:56.9550761Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9551940Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9552261Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9553185Z next(self.gen) 2025-10-10T02:02:56.9553680Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9553859Z wait_for_engine_startup( 2025-10-10T02:02:56.9553965Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9553970Z 2025-10-10T02:02:56.9556469Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9556890Z "See root cause above. " 2025-10-10T02:02:56.9557259Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9557646Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9557653Z 2025-10-10T02:02:56.9557973Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9558248Z ______ test_can_initialize_large_subset[Gemma3nForConditionalGeneration] _______ 2025-10-10T02:02:56.9558252Z 2025-10-10T02:02:56.9558361Z model_arch = 'Gemma3nForConditionalGeneration' 2025-10-10T02:02:56.9558614Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa622c30> 2025-10-10T02:02:56.9558654Z 2025-10-10T02:02:56.9563971Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9564581Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9564824Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9565210Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9565322Z  2025-10-10T02:02:56.9565706Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9565945Z  test. 2025-10-10T02:02:56.9566120Z  """ 2025-10-10T02:02:56.9566504Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9567184Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9567480Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9567487Z 2025-10-10T02:02:56.9567675Z models/test_initialization.py:131: 2025-10-10T02:02:56.9567802Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9567946Z utils.py:892: in wrapper 2025-10-10T02:02:56.9568294Z raise original_exception 2025-10-10T02:02:56.9568487Z utils.py:836: in wrapper 2025-10-10T02:02:56.9568643Z func(*args, **kwargs) 2025-10-10T02:02:56.9568867Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9569002Z LLM( 2025-10-10T02:02:56.9569584Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9569871Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9570337Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9570609Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9571203Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9572279Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9572907Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9574080Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9574446Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9574982Z super().__init__( 2025-10-10T02:02:56.9575580Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9576437Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9576755Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9577535Z next(self.gen) 2025-10-10T02:02:56.9577998Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9578296Z wait_for_engine_startup( 2025-10-10T02:02:56.9578469Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9578532Z 2025-10-10T02:02:56.9580986Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9581338Z "See root cause above. " 2025-10-10T02:02:56.9581711Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9582163Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9582168Z 2025-10-10T02:02:56.9582493Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9582763Z _______ test_can_initialize_large_subset[QwenVLForConditionalGeneration] _______ 2025-10-10T02:02:56.9582769Z 2025-10-10T02:02:56.9582881Z model_arch = 'QwenVLForConditionalGeneration' 2025-10-10T02:02:56.9583097Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6774d0> 2025-10-10T02:02:56.9583101Z 2025-10-10T02:02:56.9588186Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9588542Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9588846Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9589076Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9589165Z  2025-10-10T02:02:56.9589427Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9589526Z  test. 2025-10-10T02:02:56.9589640Z  """ 2025-10-10T02:02:56.9589940Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9590416Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9590660Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9590665Z 2025-10-10T02:02:56.9590802Z models/test_initialization.py:131: 2025-10-10T02:02:56.9590903Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9591015Z utils.py:892: in wrapper 2025-10-10T02:02:56.9591238Z raise original_exception 2025-10-10T02:02:56.9591429Z utils.py:836: in wrapper 2025-10-10T02:02:56.9591558Z func(*args, **kwargs) 2025-10-10T02:02:56.9591742Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9591866Z LLM( 2025-10-10T02:02:56.9592187Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9593720Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9594175Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9594842Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9595183Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9596853Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9597278Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9598740Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9599309Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9599725Z super().__init__( 2025-10-10T02:02:56.9606449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9607116Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9607583Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9607867Z next(self.gen) 2025-10-10T02:02:56.9608380Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9608845Z wait_for_engine_startup( 2025-10-10T02:02:56.9609073Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9609081Z 2025-10-10T02:02:56.9609837Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9610320Z "See root cause above. " 2025-10-10T02:02:56.9610795Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9611285Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9611295Z 2025-10-10T02:02:56.9611700Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9612000Z ______________ test_can_initialize_large_subset[LlamaForCausalLM] ______________ 2025-10-10T02:02:56.9612005Z 2025-10-10T02:02:56.9612105Z model_arch = 'LlamaForCausalLM' 2025-10-10T02:02:56.9612325Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa620140> 2025-10-10T02:02:56.9612332Z 2025-10-10T02:02:56.9613724Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9614070Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9614266Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9614592Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9614693Z  2025-10-10T02:02:56.9614957Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9615057Z  test. 2025-10-10T02:02:56.9615241Z  """ 2025-10-10T02:02:56.9615570Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9616009Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9616256Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9616261Z 2025-10-10T02:02:56.9616404Z models/test_initialization.py:131: 2025-10-10T02:02:56.9616580Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9616758Z utils.py:892: in wrapper 2025-10-10T02:02:56.9617053Z raise original_exception 2025-10-10T02:02:56.9617172Z utils.py:836: in wrapper 2025-10-10T02:02:56.9617303Z func(*args, **kwargs) 2025-10-10T02:02:56.9617490Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9617657Z LLM( 2025-10-10T02:02:56.9618080Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9619125Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9619608Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9620369Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9620834Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9622149Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9622592Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9623692Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9624161Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9624826Z super().__init__( 2025-10-10T02:02:56.9625260Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9626316Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9626636Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9627589Z next(self.gen) 2025-10-10T02:02:56.9628069Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9628352Z wait_for_engine_startup( 2025-10-10T02:02:56.9628453Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9628464Z 2025-10-10T02:02:56.9630926Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9631179Z "See root cause above. " 2025-10-10T02:02:56.9631851Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9632279Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9632293Z 2025-10-10T02:02:56.9632625Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9632861Z ________________ test_can_initialize_large_subset[MedusaModel] _________________ 2025-10-10T02:02:56.9632867Z 2025-10-10T02:02:56.9632941Z model_arch = 'MedusaModel' 2025-10-10T02:02:56.9633117Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63b0b0> 2025-10-10T02:02:56.9633120Z 2025-10-10T02:02:56.9638460Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9638882Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9639329Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9639681Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9639776Z  2025-10-10T02:02:56.9640040Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9640205Z  test. 2025-10-10T02:02:56.9640372Z  """ 2025-10-10T02:02:56.9640784Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9641330Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9641709Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9641722Z 2025-10-10T02:02:56.9641986Z models/test_initialization.py:131: 2025-10-10T02:02:56.9642163Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9642295Z utils.py:892: in wrapper 2025-10-10T02:02:56.9642464Z raise original_exception 2025-10-10T02:02:56.9642576Z utils.py:836: in wrapper 2025-10-10T02:02:56.9642785Z func(*args, **kwargs) 2025-10-10T02:02:56.9642999Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9643109Z LLM( 2025-10-10T02:02:56.9643573Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9644176Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9644728Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9645430Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9645883Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9647209Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9647724Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9648788Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9649243Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9649981Z super().__init__( 2025-10-10T02:02:56.9650435Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9651637Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9651990Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9652759Z next(self.gen) 2025-10-10T02:02:56.9653220Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9653704Z wait_for_engine_startup( 2025-10-10T02:02:56.9653818Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9653823Z 2025-10-10T02:02:56.9656231Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9656725Z "See root cause above. " 2025-10-10T02:02:56.9657159Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9657555Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9657560Z 2025-10-10T02:02:56.9657888Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9658204Z _______ test_can_initialize_large_subset[Glm4vForConditionalGeneration] ________ 2025-10-10T02:02:56.9658209Z 2025-10-10T02:02:56.9658313Z model_arch = 'Glm4vForConditionalGeneration' 2025-10-10T02:02:56.9658485Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa637c20> 2025-10-10T02:02:56.9658489Z 2025-10-10T02:02:56.9663729Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9664287Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9664510Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9664740Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9664835Z  2025-10-10T02:02:56.9665091Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9665189Z  test. 2025-10-10T02:02:56.9665303Z  """ 2025-10-10T02:02:56.9665763Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9666116Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9666473Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9666481Z 2025-10-10T02:02:56.9666694Z models/test_initialization.py:131: 2025-10-10T02:02:56.9666797Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9666911Z utils.py:892: in wrapper 2025-10-10T02:02:56.9667091Z raise original_exception 2025-10-10T02:02:56.9667199Z utils.py:836: in wrapper 2025-10-10T02:02:56.9667323Z func(*args, **kwargs) 2025-10-10T02:02:56.9667497Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9667594Z LLM( 2025-10-10T02:02:56.9667954Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9669567Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9669954Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9670367Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9670741Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9672166Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9672619Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9673982Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9674418Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9675027Z super().__init__( 2025-10-10T02:02:56.9675534Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9676812Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9677072Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9677956Z next(self.gen) 2025-10-10T02:02:56.9678336Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9678586Z wait_for_engine_startup( 2025-10-10T02:02:56.9678686Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9678691Z 2025-10-10T02:02:56.9681691Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9682027Z "See root cause above. " 2025-10-10T02:02:56.9682387Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9682783Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9682788Z 2025-10-10T02:02:56.9683115Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9683370Z ____________ test_can_initialize_large_subset[InternLMForCausalLM] _____________ 2025-10-10T02:02:56.9683374Z 2025-10-10T02:02:56.9683463Z model_arch = 'InternLMForCausalLM' 2025-10-10T02:02:56.9683640Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6232c0> 2025-10-10T02:02:56.9683646Z 2025-10-10T02:02:56.9688555Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9689052Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9689256Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9689483Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9689573Z  2025-10-10T02:02:56.9689826Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9689927Z  test. 2025-10-10T02:02:56.9690034Z  """ 2025-10-10T02:02:56.9690425Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9690833Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9691070Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9691075Z 2025-10-10T02:02:56.9691221Z models/test_initialization.py:131: 2025-10-10T02:02:56.9691369Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9691580Z utils.py:892: in wrapper 2025-10-10T02:02:56.9691768Z raise original_exception 2025-10-10T02:02:56.9691875Z utils.py:836: in wrapper 2025-10-10T02:02:56.9692004Z func(*args, **kwargs) 2025-10-10T02:02:56.9692257Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9692537Z LLM( 2025-10-10T02:02:56.9692950Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9694265Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9694737Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9695376Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9695753Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9697200Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9697692Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9699127Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9699593Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9700061Z super().__init__( 2025-10-10T02:02:56.9700564Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9701894Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9702162Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9702890Z next(self.gen) 2025-10-10T02:02:56.9703249Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9703586Z wait_for_engine_startup( 2025-10-10T02:02:56.9703690Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9703698Z 2025-10-10T02:02:56.9706194Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9706603Z "See root cause above. " 2025-10-10T02:02:56.9707018Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9707403Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9707408Z 2025-10-10T02:02:56.9707743Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9707987Z _______________ test_can_initialize_large_subset[NomicBertModel] _______________ 2025-10-10T02:02:56.9707993Z 2025-10-10T02:02:56.9708072Z model_arch = 'NomicBertModel' 2025-10-10T02:02:56.9708245Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa637140> 2025-10-10T02:02:56.9708250Z 2025-10-10T02:02:56.9713698Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9714028Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9714206Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9714434Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9714573Z  2025-10-10T02:02:56.9715008Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9715268Z  test. 2025-10-10T02:02:56.9715381Z  """ 2025-10-10T02:02:56.9715688Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9716025Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9716481Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9716487Z 2025-10-10T02:02:56.9716634Z models/test_initialization.py:131: 2025-10-10T02:02:56.9716734Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9716850Z utils.py:892: in wrapper 2025-10-10T02:02:56.9717012Z raise original_exception 2025-10-10T02:02:56.9717150Z utils.py:836: in wrapper 2025-10-10T02:02:56.9717374Z func(*args, **kwargs) 2025-10-10T02:02:56.9717624Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9717843Z LLM( 2025-10-10T02:02:56.9718166Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9720005Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9720519Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9721211Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9721560Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9722845Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9723208Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9724579Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9725022Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9725721Z super().__init__( 2025-10-10T02:02:56.9726056Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9727369Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9727660Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9728502Z next(self.gen) 2025-10-10T02:02:56.9728984Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9729253Z wait_for_engine_startup( 2025-10-10T02:02:56.9729353Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9729358Z 2025-10-10T02:02:56.9731709Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9732054Z "See root cause above. " 2025-10-10T02:02:56.9732417Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9733009Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9733050Z 2025-10-10T02:02:56.9733393Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9733654Z ___________ test_can_initialize_large_subset[FalconMambaForCausalLM] ___________ 2025-10-10T02:02:56.9733658Z 2025-10-10T02:02:56.9733746Z model_arch = 'FalconMambaForCausalLM' 2025-10-10T02:02:56.9733930Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6760f0> 2025-10-10T02:02:56.9733976Z 2025-10-10T02:02:56.9739589Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9739948Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9740133Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9740527Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9740678Z  2025-10-10T02:02:56.9741015Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9741127Z  test. 2025-10-10T02:02:56.9741239Z  """ 2025-10-10T02:02:56.9741547Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9742091Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9742355Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9742361Z 2025-10-10T02:02:56.9742524Z models/test_initialization.py:131: 2025-10-10T02:02:56.9742705Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9742933Z utils.py:892: in wrapper 2025-10-10T02:02:56.9743120Z raise original_exception 2025-10-10T02:02:56.9743234Z utils.py:836: in wrapper 2025-10-10T02:02:56.9743357Z func(*args, **kwargs) 2025-10-10T02:02:56.9743530Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9743631Z LLM( 2025-10-10T02:02:56.9743950Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9745277Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9745872Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9746528Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9746954Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9747963Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9748340Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9749689Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9750179Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9750639Z super().__init__( 2025-10-10T02:02:56.9751078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9752146Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9752617Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9753460Z next(self.gen) 2025-10-10T02:02:56.9753844Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9753985Z wait_for_engine_startup( 2025-10-10T02:02:56.9754084Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9754150Z 2025-10-10T02:02:56.9756581Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9757012Z "See root cause above. " 2025-10-10T02:02:56.9757442Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9757891Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9757897Z 2025-10-10T02:02:56.9758232Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9758507Z _____ test_can_initialize_large_subset[AyaVisionForConditionalGeneration] ______ 2025-10-10T02:02:56.9758514Z 2025-10-10T02:02:56.9758641Z model_arch = 'AyaVisionForConditionalGeneration' 2025-10-10T02:02:56.9758816Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6358e0> 2025-10-10T02:02:56.9758820Z 2025-10-10T02:02:56.9764405Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9764830Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9765171Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9765525Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9765636Z  2025-10-10T02:02:56.9765966Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9766094Z  test. 2025-10-10T02:02:56.9766231Z  """ 2025-10-10T02:02:56.9766669Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9767205Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9767493Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9767501Z 2025-10-10T02:02:56.9767805Z models/test_initialization.py:131: 2025-10-10T02:02:56.9767939Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9768083Z utils.py:892: in wrapper 2025-10-10T02:02:56.9768290Z raise original_exception 2025-10-10T02:02:56.9768426Z utils.py:836: in wrapper 2025-10-10T02:02:56.9768580Z func(*args, **kwargs) 2025-10-10T02:02:56.9768803Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9768923Z LLM( 2025-10-10T02:02:56.9769459Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9770176Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9770851Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9771521Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9771931Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9773160Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9773682Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9774764Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9775201Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9775863Z super().__init__( 2025-10-10T02:02:56.9776204Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9777354Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9777611Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9778546Z next(self.gen) 2025-10-10T02:02:56.9779029Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9779556Z wait_for_engine_startup( 2025-10-10T02:02:56.9779658Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9779670Z 2025-10-10T02:02:56.9781950Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9782393Z "See root cause above. " 2025-10-10T02:02:56.9782776Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9783153Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9783159Z 2025-10-10T02:02:56.9783485Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9783727Z ____________ test_can_initialize_large_subset[FalconH1ForCausalLM] _____________ 2025-10-10T02:02:56.9783731Z 2025-10-10T02:02:56.9783816Z model_arch = 'FalconH1ForCausalLM' 2025-10-10T02:02:56.9783986Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa63afc0> 2025-10-10T02:02:56.9783990Z 2025-10-10T02:02:56.9789150Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9789485Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9789704Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9790113Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9790216Z  2025-10-10T02:02:56.9790482Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9790576Z  test. 2025-10-10T02:02:56.9790700Z  """ 2025-10-10T02:02:56.9790988Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9791514Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9791910Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9791915Z 2025-10-10T02:02:56.9792052Z models/test_initialization.py:131: 2025-10-10T02:02:56.9792203Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9792491Z utils.py:892: in wrapper 2025-10-10T02:02:56.9792720Z raise original_exception 2025-10-10T02:02:56.9792829Z utils.py:836: in wrapper 2025-10-10T02:02:56.9792955Z func(*args, **kwargs) 2025-10-10T02:02:56.9793137Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9793234Z LLM( 2025-10-10T02:02:56.9793558Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9794947Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9795620Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9796410Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9796889Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9797864Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9798220Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9799652Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9800203Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9800714Z super().__init__( 2025-10-10T02:02:56.9801196Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9802036Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9802447Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9803345Z next(self.gen) 2025-10-10T02:02:56.9803794Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9804088Z wait_for_engine_startup( 2025-10-10T02:02:56.9804199Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9804212Z 2025-10-10T02:02:56.9806519Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9806825Z "See root cause above. " 2025-10-10T02:02:56.9807335Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9807737Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9807741Z 2025-10-10T02:02:56.9808078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9808328Z _______________ test_can_initialize_large_subset[MPTForCausalLM] _______________ 2025-10-10T02:02:56.9808332Z 2025-10-10T02:02:56.9808535Z model_arch = 'MPTForCausalLM' 2025-10-10T02:02:56.9808785Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6b3f20> 2025-10-10T02:02:56.9808790Z 2025-10-10T02:02:56.9813356Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9813793Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9814132Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9814366Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9814451Z  2025-10-10T02:02:56.9814723Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9814823Z  test. 2025-10-10T02:02:56.9814938Z  """ 2025-10-10T02:02:56.9815382Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9815832Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9816066Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9816074Z 2025-10-10T02:02:56.9816210Z models/test_initialization.py:131: 2025-10-10T02:02:56.9816313Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9816431Z utils.py:892: in wrapper 2025-10-10T02:02:56.9816603Z raise original_exception 2025-10-10T02:02:56.9816713Z utils.py:836: in wrapper 2025-10-10T02:02:56.9816829Z func(*args, **kwargs) 2025-10-10T02:02:56.9817011Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9817124Z LLM( 2025-10-10T02:02:56.9817449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9818813Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9819283Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9820169Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9820648Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9821744Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9822093Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9823402Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9823744Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9824317Z super().__init__( 2025-10-10T02:02:56.9824642Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9825585Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9825824Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9826837Z next(self.gen) 2025-10-10T02:02:56.9827206Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9827568Z wait_for_engine_startup( 2025-10-10T02:02:56.9827720Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9827727Z 2025-10-10T02:02:56.9830089Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9830430Z "See root cause above. " 2025-10-10T02:02:56.9830881Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9831261Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9831265Z 2025-10-10T02:02:56.9831599Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9831830Z ___________________ test_can_initialize_large_subset[NVLM_D] ___________________ 2025-10-10T02:02:56.9831834Z 2025-10-10T02:02:56.9831953Z model_arch = 'NVLM_D' 2025-10-10T02:02:56.9832129Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa637bc0> 2025-10-10T02:02:56.9832133Z 2025-10-10T02:02:56.9837188Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9837587Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9837779Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9838104Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9838251Z  2025-10-10T02:02:56.9838622Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9838730Z  test. 2025-10-10T02:02:56.9838841Z  """ 2025-10-10T02:02:56.9839335Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9839729Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9840048Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9840056Z 2025-10-10T02:02:56.9840229Z models/test_initialization.py:131: 2025-10-10T02:02:56.9840333Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9840446Z utils.py:892: in wrapper 2025-10-10T02:02:56.9840615Z raise original_exception 2025-10-10T02:02:56.9840746Z utils.py:836: in wrapper 2025-10-10T02:02:56.9840966Z func(*args, **kwargs) 2025-10-10T02:02:56.9841178Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9841279Z LLM( 2025-10-10T02:02:56.9841601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9843002Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9843529Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9844303Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9844688Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9845846Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9846476Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9847134Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9847524Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9848209Z super().__init__( 2025-10-10T02:02:56.9848563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9849577Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9849815Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9850690Z next(self.gen) 2025-10-10T02:02:56.9851064Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9851707Z wait_for_engine_startup( 2025-10-10T02:02:56.9851824Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9851829Z 2025-10-10T02:02:56.9854074Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9854470Z "See root cause above. " 2025-10-10T02:02:56.9854846Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9855244Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9855252Z 2025-10-10T02:02:56.9855593Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9855838Z ______________ test_can_initialize_large_subset[GPT2LMHeadModel] _______________ 2025-10-10T02:02:56.9855842Z 2025-10-10T02:02:56.9855928Z model_arch = 'GPT2LMHeadModel' 2025-10-10T02:02:56.9856107Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6b31d0> 2025-10-10T02:02:56.9856111Z 2025-10-10T02:02:56.9861371Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9861852Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9862056Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9862288Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9862377Z  2025-10-10T02:02:56.9862651Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9862750Z  test. 2025-10-10T02:02:56.9862858Z  """ 2025-10-10T02:02:56.9863308Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9863678Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9864042Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9864050Z 2025-10-10T02:02:56.9864207Z models/test_initialization.py:131: 2025-10-10T02:02:56.9864470Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9864721Z utils.py:892: in wrapper 2025-10-10T02:02:56.9864959Z raise original_exception 2025-10-10T02:02:56.9865078Z utils.py:836: in wrapper 2025-10-10T02:02:56.9865204Z func(*args, **kwargs) 2025-10-10T02:02:56.9865524Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9865887Z LLM( 2025-10-10T02:02:56.9866214Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9866546Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9867063Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9867696Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9868062Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9869158Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9869597Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9870644Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9871029Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9871667Z super().__init__( 2025-10-10T02:02:56.9872000Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9873025Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9873261Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9874133Z next(self.gen) 2025-10-10T02:02:56.9874502Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9875031Z wait_for_engine_startup( 2025-10-10T02:02:56.9875145Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9875150Z 2025-10-10T02:02:56.9877336Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9877699Z "See root cause above. " 2025-10-10T02:02:56.9878187Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9878593Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9878599Z 2025-10-10T02:02:56.9879122Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9879410Z ________ test_can_initialize_large_subset[AriaForConditionalGeneration] ________ 2025-10-10T02:02:56.9879419Z 2025-10-10T02:02:56.9879519Z model_arch = 'AriaForConditionalGeneration' 2025-10-10T02:02:56.9879694Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa677740> 2025-10-10T02:02:56.9879705Z 2025-10-10T02:02:56.9884462Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9884940Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9885177Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9885398Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9885492Z  2025-10-10T02:02:56.9885745Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9885979Z  test. 2025-10-10T02:02:56.9886133Z  """ 2025-10-10T02:02:56.9886433Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9886890Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9887159Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9887167Z 2025-10-10T02:02:56.9887345Z models/test_initialization.py:131: 2025-10-10T02:02:56.9887591Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9887724Z utils.py:892: in wrapper 2025-10-10T02:02:56.9887898Z raise original_exception 2025-10-10T02:02:56.9888007Z utils.py:836: in wrapper 2025-10-10T02:02:56.9888147Z func(*args, **kwargs) 2025-10-10T02:02:56.9888466Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9888571Z LLM( 2025-10-10T02:02:56.9888901Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9890130Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9890655Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9891414Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9891810Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9892799Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9893212Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9894560Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9895098Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9895427Z super().__init__( 2025-10-10T02:02:56.9895761Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9897440Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9897681Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9898442Z next(self.gen) 2025-10-10T02:02:56.9898838Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9898981Z wait_for_engine_startup( 2025-10-10T02:02:56.9899082Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9899087Z 2025-10-10T02:02:56.9901524Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9901924Z "See root cause above. " 2025-10-10T02:02:56.9902390Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9902780Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9902843Z 2025-10-10T02:02:56.9903189Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9903438Z _______________ test_can_initialize_large_subset[OPTForCausalLM] _______________ 2025-10-10T02:02:56.9903442Z 2025-10-10T02:02:56.9903525Z model_arch = 'OPTForCausalLM' 2025-10-10T02:02:56.9903701Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6b31d0> 2025-10-10T02:02:56.9903705Z 2025-10-10T02:02:56.9908586Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9909038Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9909379Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9909638Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9909734Z  2025-10-10T02:02:56.9909997Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9910098Z  test. 2025-10-10T02:02:56.9910208Z  """ 2025-10-10T02:02:56.9910519Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9911004Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9911247Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9911252Z 2025-10-10T02:02:56.9911414Z models/test_initialization.py:131: 2025-10-10T02:02:56.9911586Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9911755Z utils.py:892: in wrapper 2025-10-10T02:02:56.9911938Z raise original_exception 2025-10-10T02:02:56.9912052Z utils.py:836: in wrapper 2025-10-10T02:02:56.9912173Z func(*args, **kwargs) 2025-10-10T02:02:56.9912350Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9912454Z LLM( 2025-10-10T02:02:56.9912783Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9914401Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9914854Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9915513Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9915994Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9916956Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9917372Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9918788Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9919315Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9919676Z super().__init__( 2025-10-10T02:02:56.9920191Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9921202Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9921542Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9922246Z next(self.gen) 2025-10-10T02:02:56.9922613Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9923026Z wait_for_engine_startup( 2025-10-10T02:02:56.9923135Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9923146Z 2025-10-10T02:02:56.9925414Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9925744Z "See root cause above. " 2025-10-10T02:02:56.9926231Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9926636Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9926641Z 2025-10-10T02:02:56.9927094Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9927363Z ______________ test_can_initialize_large_subset[BambaForCausalLM] ______________ 2025-10-10T02:02:56.9927371Z 2025-10-10T02:02:56.9927455Z model_arch = 'BambaForCausalLM' 2025-10-10T02:02:56.9927633Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa637e90> 2025-10-10T02:02:56.9927639Z 2025-10-10T02:02:56.9932964Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9933431Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9933737Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9933977Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9934064Z  2025-10-10T02:02:56.9934329Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9934511Z  test. 2025-10-10T02:02:56.9934671Z  """ 2025-10-10T02:02:56.9935049Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9935453Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9935674Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9935686Z 2025-10-10T02:02:56.9935873Z models/test_initialization.py:131: 2025-10-10T02:02:56.9936044Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9936174Z utils.py:892: in wrapper 2025-10-10T02:02:56.9936338Z raise original_exception 2025-10-10T02:02:56.9936447Z utils.py:836: in wrapper 2025-10-10T02:02:56.9936570Z func(*args, **kwargs) 2025-10-10T02:02:56.9936900Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9937082Z LLM( 2025-10-10T02:02:56.9937491Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9937860Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9938254Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9939174Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9939579Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9940640Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9941049Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9942336Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9942894Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9943347Z super().__init__( 2025-10-10T02:02:56.9943701Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9944743Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9944985Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9945835Z next(self.gen) 2025-10-10T02:02:56.9946210Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9946622Z wait_for_engine_startup( 2025-10-10T02:02:56.9946734Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9946743Z 2025-10-10T02:02:56.9949121Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9949438Z "See root cause above. " 2025-10-10T02:02:56.9949803Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9950183Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9950188Z 2025-10-10T02:02:56.9950523Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9950805Z ___ test_can_initialize_large_subset[Cohere2VisionForConditionalGeneration] ____ 2025-10-10T02:02:56.9950812Z 2025-10-10T02:02:56.9950941Z model_arch = 'Cohere2VisionForConditionalGeneration' 2025-10-10T02:02:56.9951120Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6ce7b0> 2025-10-10T02:02:56.9951124Z 2025-10-10T02:02:56.9956133Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9956493Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9956814Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9957057Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9957215Z  2025-10-10T02:02:56.9957515Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9957616Z  test. 2025-10-10T02:02:56.9957724Z  """ 2025-10-10T02:02:56.9958093Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9958538Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9958829Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9958834Z 2025-10-10T02:02:56.9959154Z models/test_initialization.py:131: 2025-10-10T02:02:56.9959336Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9959462Z utils.py:892: in wrapper 2025-10-10T02:02:56.9959632Z raise original_exception 2025-10-10T02:02:56.9959745Z utils.py:836: in wrapper 2025-10-10T02:02:56.9959939Z func(*args, **kwargs) 2025-10-10T02:02:56.9960127Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9960299Z LLM( 2025-10-10T02:02:56.9960690Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9962248Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9962640Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9963251Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9963764Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9964890Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9965396Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9966564Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9966916Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9967403Z super().__init__( 2025-10-10T02:02:56.9967912Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9969176Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9969422Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9970137Z next(self.gen) 2025-10-10T02:02:56.9970501Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9970703Z wait_for_engine_startup( 2025-10-10T02:02:56.9970806Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9970814Z 2025-10-10T02:02:56.9973414Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9973686Z "See root cause above. " 2025-10-10T02:02:56.9974051Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9974510Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9974549Z 2025-10-10T02:02:56.9974890Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9975163Z __________ test_can_initialize_large_subset[HunYuanMoEV1ForCausalLM] ___________ 2025-10-10T02:02:56.9975175Z 2025-10-10T02:02:56.9975266Z model_arch = 'HunYuanMoEV1ForCausalLM' 2025-10-10T02:02:56.9975474Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa637770> 2025-10-10T02:02:56.9975477Z 2025-10-10T02:02:56.9980292Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:56.9980620Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:56.9980810Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:56.9981108Z  """Test initializing large subset of supported models 2025-10-10T02:02:56.9981327Z  2025-10-10T02:02:56.9981613Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:56.9981713Z  test. 2025-10-10T02:02:56.9981826Z  """ 2025-10-10T02:02:56.9982117Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:56.9982455Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:56.9982773Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:56.9982792Z 2025-10-10T02:02:56.9982966Z models/test_initialization.py:131: 2025-10-10T02:02:56.9983072Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9983182Z utils.py:892: in wrapper 2025-10-10T02:02:56.9983351Z raise original_exception 2025-10-10T02:02:56.9983459Z utils.py:836: in wrapper 2025-10-10T02:02:56.9983614Z func(*args, **kwargs) 2025-10-10T02:02:56.9983889Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:56.9984438Z LLM( 2025-10-10T02:02:56.9984769Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:56.9986147Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:56.9986719Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:56.9987503Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:56.9987919Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:56.9988946Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:56.9989296Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:56.9990887Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:56.9991364Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:56.9991750Z super().__init__( 2025-10-10T02:02:56.9992195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:56.9993260Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:56.9993760Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:56.9994538Z next(self.gen) 2025-10-10T02:02:56.9994917Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:56.9995246Z wait_for_engine_startup( 2025-10-10T02:02:56.9995374Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:56.9995380Z 2025-10-10T02:02:56.9997795Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:56.9998183Z "See root cause above. " 2025-10-10T02:02:56.9998560Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:56.9999281Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:56.9999292Z 2025-10-10T02:02:56.9999649Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:56.9999923Z __________ test_can_initialize_large_subset[Fairseq2LlamaForCausalLM] __________ 2025-10-10T02:02:56.9999927Z 2025-10-10T02:02:57.0000020Z model_arch = 'Fairseq2LlamaForCausalLM' 2025-10-10T02:02:57.0000196Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6cfe30> 2025-10-10T02:02:57.0000200Z 2025-10-10T02:02:57.0005277Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0005745Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0005941Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0006182Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0006270Z  2025-10-10T02:02:57.0006661Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0006786Z  test. 2025-10-10T02:02:57.0006984Z  """ 2025-10-10T02:02:57.0007342Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0007710Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0008064Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0008070Z 2025-10-10T02:02:57.0008225Z models/test_initialization.py:131: 2025-10-10T02:02:57.0008324Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0008443Z utils.py:892: in wrapper 2025-10-10T02:02:57.0008612Z raise original_exception 2025-10-10T02:02:57.0008728Z utils.py:836: in wrapper 2025-10-10T02:02:57.0008846Z func(*args, **kwargs) 2025-10-10T02:02:57.0009138Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0009335Z LLM( 2025-10-10T02:02:57.0009729Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0010793Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0011251Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0011866Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0012378Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0013367Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0014000Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0015974Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0016469Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0016667Z super().__init__( 2025-10-10T02:02:57.0016999Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0017567Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0017809Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0018697Z next(self.gen) 2025-10-10T02:02:57.0019058Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0019346Z wait_for_engine_startup( 2025-10-10T02:02:57.0019517Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0019525Z 2025-10-10T02:02:57.0021865Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0022255Z "See root cause above. " 2025-10-10T02:02:57.0022625Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0023137Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0023147Z 2025-10-10T02:02:57.0023490Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0023750Z __________ test_can_initialize_large_subset[TransformersForCausalLM] ___________ 2025-10-10T02:02:57.0023755Z 2025-10-10T02:02:57.0023847Z model_arch = 'TransformersForCausalLM' 2025-10-10T02:02:57.0024029Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6b0260> 2025-10-10T02:02:57.0024036Z 2025-10-10T02:02:57.0028989Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0029396Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0029620Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0029943Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0030035Z  2025-10-10T02:02:57.0030296Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0030393Z  test. 2025-10-10T02:02:57.0030502Z  """ 2025-10-10T02:02:57.0030869Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0031282Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0031609Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0031614Z 2025-10-10T02:02:57.0031759Z models/test_initialization.py:131: 2025-10-10T02:02:57.0031861Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0032145Z utils.py:892: in wrapper 2025-10-10T02:02:57.0032383Z raise original_exception 2025-10-10T02:02:57.0032561Z utils.py:836: in wrapper 2025-10-10T02:02:57.0032687Z func(*args, **kwargs) 2025-10-10T02:02:57.0032876Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0033212Z LLM( 2025-10-10T02:02:57.0033829Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0034902Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0035306Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0036140Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0036497Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0037842Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0038203Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0039610Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0039977Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0040672Z super().__init__( 2025-10-10T02:02:57.0041016Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0042239Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0042480Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0043210Z next(self.gen) 2025-10-10T02:02:57.0043587Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0043961Z wait_for_engine_startup( 2025-10-10T02:02:57.0044096Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0044111Z 2025-10-10T02:02:57.0046339Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0046649Z "See root cause above. " 2025-10-10T02:02:57.0047094Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0047583Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0047588Z 2025-10-10T02:02:57.0047990Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0048381Z ____________ test_can_initialize_large_subset[SkyworkR1VChatModel] _____________ 2025-10-10T02:02:57.0048431Z 2025-10-10T02:02:57.0048538Z model_arch = 'SkyworkR1VChatModel' 2025-10-10T02:02:57.0048758Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6cc110> 2025-10-10T02:02:57.0048768Z 2025-10-10T02:02:57.0053685Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0054023Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0054289Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0054514Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0054607Z  2025-10-10T02:02:57.0054860Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0054967Z  test. 2025-10-10T02:02:57.0055076Z  """ 2025-10-10T02:02:57.0055418Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0055774Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0055994Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0056000Z 2025-10-10T02:02:57.0056145Z models/test_initialization.py:131: 2025-10-10T02:02:57.0056245Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0056355Z utils.py:892: in wrapper 2025-10-10T02:02:57.0056525Z raise original_exception 2025-10-10T02:02:57.0056630Z utils.py:836: in wrapper 2025-10-10T02:02:57.0057226Z func(*args, **kwargs) 2025-10-10T02:02:57.0057408Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0058169Z LLM( 2025-10-10T02:02:57.0058536Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0059958Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0060374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0061088Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0061449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0062749Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0063131Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0064457Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0064842Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0065307Z super().__init__( 2025-10-10T02:02:57.0065692Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0066649Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0066902Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0067683Z next(self.gen) 2025-10-10T02:02:57.0068291Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0068561Z wait_for_engine_startup( 2025-10-10T02:02:57.0068670Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0068676Z 2025-10-10T02:02:57.0070903Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0071218Z "See root cause above. " 2025-10-10T02:02:57.0071588Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0071976Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0071981Z 2025-10-10T02:02:57.0072374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0073126Z ____________ test_can_initialize_large_subset[InternLM2ForCausalLM] ____________ 2025-10-10T02:02:57.0073134Z 2025-10-10T02:02:57.0073239Z model_arch = 'InternLM2ForCausalLM' 2025-10-10T02:02:57.0073419Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa622ff0> 2025-10-10T02:02:57.0073423Z 2025-10-10T02:02:57.0078571Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0078903Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0079194Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0079436Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0079527Z  2025-10-10T02:02:57.0079795Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0079894Z  test. 2025-10-10T02:02:57.0080009Z  """ 2025-10-10T02:02:57.0080303Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0080643Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0080870Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0080874Z 2025-10-10T02:02:57.0081013Z models/test_initialization.py:131: 2025-10-10T02:02:57.0081120Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0081241Z utils.py:892: in wrapper 2025-10-10T02:02:57.0081411Z raise original_exception 2025-10-10T02:02:57.0081524Z utils.py:836: in wrapper 2025-10-10T02:02:57.0082261Z func(*args, **kwargs) 2025-10-10T02:02:57.0083000Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0083102Z LLM( 2025-10-10T02:02:57.0083442Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0084703Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0085065Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0085924Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0086329Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0087557Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0087922Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0089180Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0089523Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0090118Z super().__init__( 2025-10-10T02:02:57.0090459Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0091582Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0091826Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0092655Z next(self.gen) 2025-10-10T02:02:57.0093073Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0093401Z wait_for_engine_startup( 2025-10-10T02:02:57.0093509Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0093517Z 2025-10-10T02:02:57.0095983Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0096429Z "See root cause above. " 2025-10-10T02:02:57.0096800Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0097185Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0097196Z 2025-10-10T02:02:57.0097525Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0097792Z ____ test_can_initialize_large_subset[Qwen2_5OmniForConditionalGeneration] _____ 2025-10-10T02:02:57.0097799Z 2025-10-10T02:02:57.0097919Z model_arch = 'Qwen2_5OmniForConditionalGeneration' 2025-10-10T02:02:57.0098089Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa636810> 2025-10-10T02:02:57.0098092Z 2025-10-10T02:02:57.0103680Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0104045Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0104281Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0104562Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0104670Z  2025-10-10T02:02:57.0104994Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0105115Z  test. 2025-10-10T02:02:57.0105258Z  """ 2025-10-10T02:02:57.0105613Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0106037Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0106311Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0106450Z 2025-10-10T02:02:57.0106629Z models/test_initialization.py:131: 2025-10-10T02:02:57.0106825Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0106969Z utils.py:892: in wrapper 2025-10-10T02:02:57.0107188Z raise original_exception 2025-10-10T02:02:57.0107327Z utils.py:836: in wrapper 2025-10-10T02:02:57.0107676Z func(*args, **kwargs) 2025-10-10T02:02:57.0108005Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0108110Z LLM( 2025-10-10T02:02:57.0108439Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0110215Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0110693Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0111058Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0111622Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0112566Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0113012Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0114113Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0114532Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0115116Z super().__init__( 2025-10-10T02:02:57.0115537Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0116530Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0116826Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0117649Z next(self.gen) 2025-10-10T02:02:57.0118089Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0118303Z wait_for_engine_startup( 2025-10-10T02:02:57.0118428Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0118434Z 2025-10-10T02:02:57.0121044Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0121349Z "See root cause above. " 2025-10-10T02:02:57.0121807Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0122279Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0122285Z 2025-10-10T02:02:57.0122717Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0123044Z ______ test_can_initialize_large_subset[ChatGLMForConditionalGeneration] _______ 2025-10-10T02:02:57.0123048Z 2025-10-10T02:02:57.0123186Z model_arch = 'ChatGLMForConditionalGeneration' 2025-10-10T02:02:57.0123402Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124286b0> 2025-10-10T02:02:57.0123406Z 2025-10-10T02:02:57.0128446Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0128849Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0129032Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0129273Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0129402Z  2025-10-10T02:02:57.0129664Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0129763Z  test. 2025-10-10T02:02:57.0129872Z  """ 2025-10-10T02:02:57.0130168Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0130586Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0130886Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0130965Z 2025-10-10T02:02:57.0131106Z models/test_initialization.py:131: 2025-10-10T02:02:57.0131204Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0131324Z utils.py:892: in wrapper 2025-10-10T02:02:57.0131497Z raise original_exception 2025-10-10T02:02:57.0131609Z utils.py:836: in wrapper 2025-10-10T02:02:57.0132027Z func(*args, **kwargs) 2025-10-10T02:02:57.0132306Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0132759Z LLM( 2025-10-10T02:02:57.0133104Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0135010Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0135611Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0135910Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0136491Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0137423Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0138002Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0139391Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0139846Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0140084Z super().__init__( 2025-10-10T02:02:57.0140768Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0141457Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0141967Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0142697Z next(self.gen) 2025-10-10T02:02:57.0143206Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0143375Z wait_for_engine_startup( 2025-10-10T02:02:57.0143505Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0143512Z 2025-10-10T02:02:57.0145812Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0146316Z "See root cause above. " 2025-10-10T02:02:57.0146878Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0147491Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0147504Z 2025-10-10T02:02:57.0147849Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0148108Z ___________ test_can_initialize_large_subset[EagleDeepSeekMTPModel] ____________ 2025-10-10T02:02:57.0148112Z 2025-10-10T02:02:57.0148204Z model_arch = 'EagleDeepSeekMTPModel' 2025-10-10T02:02:57.0148381Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6ccaa0> 2025-10-10T02:02:57.0148387Z 2025-10-10T02:02:57.0153794Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0154146Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0154360Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0154777Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0154891Z  2025-10-10T02:02:57.0155159Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0155259Z  test. 2025-10-10T02:02:57.0155375Z  """ 2025-10-10T02:02:57.0155674Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0156083Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0156354Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0156359Z 2025-10-10T02:02:57.0156496Z models/test_initialization.py:131: 2025-10-10T02:02:57.0156605Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0156717Z utils.py:892: in wrapper 2025-10-10T02:02:57.0156970Z raise original_exception 2025-10-10T02:02:57.0157086Z utils.py:836: in wrapper 2025-10-10T02:02:57.0157206Z func(*args, **kwargs) 2025-10-10T02:02:57.0157392Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0157696Z LLM( 2025-10-10T02:02:57.0158123Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0159631Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0160116Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0160737Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0161153Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0162179Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0162608Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0163817Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0164208Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0164768Z super().__init__( 2025-10-10T02:02:57.0165107Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0166125Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0166374Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0167510Z next(self.gen) 2025-10-10T02:02:57.0167880Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0168117Z wait_for_engine_startup( 2025-10-10T02:02:57.0168234Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0168239Z 2025-10-10T02:02:57.0170641Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0170900Z "See root cause above. " 2025-10-10T02:02:57.0171365Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0171758Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0171762Z 2025-10-10T02:02:57.0172087Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0172350Z ____________ test_can_initialize_large_subset[PersimmonForCausalLM] ____________ 2025-10-10T02:02:57.0172361Z 2025-10-10T02:02:57.0172452Z model_arch = 'PersimmonForCausalLM' 2025-10-10T02:02:57.0172626Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6b09b0> 2025-10-10T02:02:57.0172630Z 2025-10-10T02:02:57.0177936Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0178282Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0178469Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0178696Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0178794Z  2025-10-10T02:02:57.0179051Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0179153Z  test. 2025-10-10T02:02:57.0179270Z  """ 2025-10-10T02:02:57.0179663Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0180013Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0180234Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0180239Z 2025-10-10T02:02:57.0180391Z models/test_initialization.py:131: 2025-10-10T02:02:57.0180488Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0180598Z utils.py:892: in wrapper 2025-10-10T02:02:57.0180761Z raise original_exception 2025-10-10T02:02:57.0180953Z utils.py:836: in wrapper 2025-10-10T02:02:57.0181816Z func(*args, **kwargs) 2025-10-10T02:02:57.0182109Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0182658Z LLM( 2025-10-10T02:02:57.0182987Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0184488Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0185025Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0185766Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0186098Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0187324Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0187687Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0189032Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0189483Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0189958Z super().__init__( 2025-10-10T02:02:57.0190296Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0191550Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0191856Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0192467Z next(self.gen) 2025-10-10T02:02:57.0192887Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0193130Z wait_for_engine_startup( 2025-10-10T02:02:57.0193233Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0193238Z 2025-10-10T02:02:57.0195651Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0195984Z "See root cause above. " 2025-10-10T02:02:57.0196575Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0196972Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0196978Z 2025-10-10T02:02:57.0197291Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0197543Z _________ test_can_initialize_large_subset[Eagle3Qwen2_5vlForCausalLM] _________ 2025-10-10T02:02:57.0197547Z 2025-10-10T02:02:57.0197632Z model_arch = 'Eagle3Qwen2_5vlForCausalLM' 2025-10-10T02:02:57.0197807Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6239b0> 2025-10-10T02:02:57.0197814Z 2025-10-10T02:02:57.0203089Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0203415Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0203593Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0203945Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0204196Z  2025-10-10T02:02:57.0204469Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0204573Z  test. 2025-10-10T02:02:57.0204689Z  """ 2025-10-10T02:02:57.0204980Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0205572Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0205895Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0205901Z 2025-10-10T02:02:57.0206045Z models/test_initialization.py:131: 2025-10-10T02:02:57.0206195Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0206409Z utils.py:892: in wrapper 2025-10-10T02:02:57.0206754Z raise original_exception 2025-10-10T02:02:57.0206876Z utils.py:836: in wrapper 2025-10-10T02:02:57.0206999Z func(*args, **kwargs) 2025-10-10T02:02:57.0207190Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0207296Z LLM( 2025-10-10T02:02:57.0207623Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0209027Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0209601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0210157Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0210570Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0211857Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0212413Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0213471Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0213915Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0214336Z super().__init__( 2025-10-10T02:02:57.0214770Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0215832Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0216252Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0217025Z next(self.gen) 2025-10-10T02:02:57.0217426Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0217563Z wait_for_engine_startup( 2025-10-10T02:02:57.0217661Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0217667Z 2025-10-10T02:02:57.0219896Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0220313Z "See root cause above. " 2025-10-10T02:02:57.0220800Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0221237Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0221242Z 2025-10-10T02:02:57.0221572Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0221852Z _____ test_can_initialize_large_subset[Qwen2AudioForConditionalGeneration] _____ 2025-10-10T02:02:57.0221903Z 2025-10-10T02:02:57.0222019Z model_arch = 'Qwen2AudioForConditionalGeneration' 2025-10-10T02:02:57.0222195Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa61f740> 2025-10-10T02:02:57.0222199Z 2025-10-10T02:02:57.0227105Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0227457Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0227749Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0228166Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0228265Z  2025-10-10T02:02:57.0228527Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0228635Z  test. 2025-10-10T02:02:57.0228743Z  """ 2025-10-10T02:02:57.0229040Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0229589Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0229842Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0229854Z 2025-10-10T02:02:57.0229993Z models/test_initialization.py:131: 2025-10-10T02:02:57.0230156Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0230372Z utils.py:892: in wrapper 2025-10-10T02:02:57.0230656Z raise original_exception 2025-10-10T02:02:57.0230777Z utils.py:836: in wrapper 2025-10-10T02:02:57.0230901Z func(*args, **kwargs) 2025-10-10T02:02:57.0231076Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0231184Z LLM( 2025-10-10T02:02:57.0231497Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0232823Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0233419Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0234002Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0234347Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0235591Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0235969Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0237128Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0237671Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0238139Z super().__init__( 2025-10-10T02:02:57.0238560Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0239753Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0240170Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0241062Z next(self.gen) 2025-10-10T02:02:57.0241486Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0241698Z wait_for_engine_startup( 2025-10-10T02:02:57.0241799Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0241805Z 2025-10-10T02:02:57.0244041Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0244344Z "See root cause above. " 2025-10-10T02:02:57.0244852Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0245252Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0245259Z 2025-10-10T02:02:57.0245588Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0245833Z _____________ test_can_initialize_large_subset[ExaoneForCausalLM] ______________ 2025-10-10T02:02:57.0245837Z 2025-10-10T02:02:57.0245917Z model_arch = 'ExaoneForCausalLM' 2025-10-10T02:02:57.0246092Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6cd190> 2025-10-10T02:02:57.0246096Z 2025-10-10T02:02:57.0251163Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0251502Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0251685Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0252022Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0252208Z  2025-10-10T02:02:57.0252519Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0252615Z  test. 2025-10-10T02:02:57.0252722Z  """ 2025-10-10T02:02:57.0253016Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0253399Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0253810Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0253816Z 2025-10-10T02:02:57.0253968Z models/test_initialization.py:131: 2025-10-10T02:02:57.0254066Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0254182Z utils.py:892: in wrapper 2025-10-10T02:02:57.0254485Z raise original_exception 2025-10-10T02:02:57.0254690Z utils.py:836: in wrapper 2025-10-10T02:02:57.0254820Z func(*args, **kwargs) 2025-10-10T02:02:57.0255000Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0255120Z LLM( 2025-10-10T02:02:57.0255530Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0257051Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0257611Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0258180Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0258598Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0259629Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0259987Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0261511Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0261932Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0262274Z super().__init__( 2025-10-10T02:02:57.0262733Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0263821Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0264255Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0265152Z next(self.gen) 2025-10-10T02:02:57.0265531Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0265830Z wait_for_engine_startup( 2025-10-10T02:02:57.0265938Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0265946Z 2025-10-10T02:02:57.0268149Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0268478Z "See root cause above. " 2025-10-10T02:02:57.0269133Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0269550Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0269555Z 2025-10-10T02:02:57.0269891Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0270137Z _______________ test_can_initialize_large_subset[RWForCausalLM] ________________ 2025-10-10T02:02:57.0270141Z 2025-10-10T02:02:57.0270220Z model_arch = 'RWForCausalLM' 2025-10-10T02:02:57.0270402Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa637fe0> 2025-10-10T02:02:57.0270405Z 2025-10-10T02:02:57.0275507Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0276106Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0276429Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0276884Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0276996Z  2025-10-10T02:02:57.0277267Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0277374Z  test. 2025-10-10T02:02:57.0277577Z  """ 2025-10-10T02:02:57.0278001Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0278590Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0279138Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0279223Z 2025-10-10T02:02:57.0279395Z models/test_initialization.py:131: 2025-10-10T02:02:57.0279507Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0279623Z utils.py:892: in wrapper 2025-10-10T02:02:57.0279786Z raise original_exception 2025-10-10T02:02:57.0279894Z utils.py:836: in wrapper 2025-10-10T02:02:57.0280011Z func(*args, **kwargs) 2025-10-10T02:02:57.0280189Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0280376Z LLM( 2025-10-10T02:02:57.0280939Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0281209Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0281565Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0282592Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0282936Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0284272Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0284638Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0285667Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0286032Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0287677Z super().__init__( 2025-10-10T02:02:57.0288040Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0288315Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0288751Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0289279Z next(self.gen) 2025-10-10T02:02:57.0289787Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0290007Z wait_for_engine_startup( 2025-10-10T02:02:57.0290116Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0290123Z 2025-10-10T02:02:57.0292665Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0293130Z "See root cause above. " 2025-10-10T02:02:57.0293546Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0294026Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0294031Z 2025-10-10T02:02:57.0294438Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0294730Z _____________ test_can_initialize_large_subset[ApertusForCausalLM] _____________ 2025-10-10T02:02:57.0294733Z 2025-10-10T02:02:57.0294825Z model_arch = 'ApertusForCausalLM' 2025-10-10T02:02:57.0295001Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6c7650> 2025-10-10T02:02:57.0295004Z 2025-10-10T02:02:57.0300227Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0300748Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0300938Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0301172Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0301257Z  2025-10-10T02:02:57.0301648Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0301769Z  test. 2025-10-10T02:02:57.0301963Z  """ 2025-10-10T02:02:57.0302270Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0302608Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0302837Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0302843Z 2025-10-10T02:02:57.0302978Z models/test_initialization.py:131: 2025-10-10T02:02:57.0303147Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0303360Z utils.py:892: in wrapper 2025-10-10T02:02:57.0303547Z raise original_exception 2025-10-10T02:02:57.0303662Z utils.py:836: in wrapper 2025-10-10T02:02:57.0303786Z func(*args, **kwargs) 2025-10-10T02:02:57.0304011Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0304484Z LLM( 2025-10-10T02:02:57.0304827Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0306427Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0306803Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0307688Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0308021Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0309406Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0309769Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0310853Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0311192Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0311948Z super().__init__( 2025-10-10T02:02:57.0312274Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0313606Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0313848Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0314949Z next(self.gen) 2025-10-10T02:02:57.0315533Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0315781Z wait_for_engine_startup( 2025-10-10T02:02:57.0315887Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0315892Z 2025-10-10T02:02:57.0318295Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0318621Z "See root cause above. " 2025-10-10T02:02:57.0319201Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0319632Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0319640Z 2025-10-10T02:02:57.0320030Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0320281Z ______________ test_can_initialize_large_subset[OlmoForCausalLM] _______________ 2025-10-10T02:02:57.0320285Z 2025-10-10T02:02:57.0320370Z model_arch = 'OlmoForCausalLM' 2025-10-10T02:02:57.0320548Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6cd6a0> 2025-10-10T02:02:57.0320551Z 2025-10-10T02:02:57.0325957Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0326346Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0326583Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0326872Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0326962Z  2025-10-10T02:02:57.0327335Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0327448Z  test. 2025-10-10T02:02:57.0327561Z  """ 2025-10-10T02:02:57.0327858Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0328335Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0328569Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0328574Z 2025-10-10T02:02:57.0328737Z models/test_initialization.py:131: 2025-10-10T02:02:57.0328924Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0329064Z utils.py:892: in wrapper 2025-10-10T02:02:57.0329237Z raise original_exception 2025-10-10T02:02:57.0329345Z utils.py:836: in wrapper 2025-10-10T02:02:57.0329470Z func(*args, **kwargs) 2025-10-10T02:02:57.0329730Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0330133Z LLM( 2025-10-10T02:02:57.0330465Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0332094Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0332466Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0333430Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0333805Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0334837Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0335183Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0336629Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0337141Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0337652Z super().__init__( 2025-10-10T02:02:57.0337997Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0338980Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0339223Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0340190Z next(self.gen) 2025-10-10T02:02:57.0340703Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0341002Z wait_for_engine_startup( 2025-10-10T02:02:57.0341118Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0341128Z 2025-10-10T02:02:57.0343275Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0343521Z "See root cause above. " 2025-10-10T02:02:57.0344077Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0344539Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0344544Z 2025-10-10T02:02:57.0344883Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0345154Z ______ test_can_initialize_large_subset[PixtralForConditionalGeneration] _______ 2025-10-10T02:02:57.0345158Z 2025-10-10T02:02:57.0345274Z model_arch = 'PixtralForConditionalGeneration' 2025-10-10T02:02:57.0345448Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa675d30> 2025-10-10T02:02:57.0345452Z 2025-10-10T02:02:57.0350539Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0351000Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0351212Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0351447Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0351564Z  2025-10-10T02:02:57.0351929Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0352036Z  test. 2025-10-10T02:02:57.0352145Z  """ 2025-10-10T02:02:57.0352450Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0352796Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0353179Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0353246Z 2025-10-10T02:02:57.0353432Z models/test_initialization.py:131: 2025-10-10T02:02:57.0353541Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0353654Z utils.py:892: in wrapper 2025-10-10T02:02:57.0353854Z raise original_exception 2025-10-10T02:02:57.0354116Z utils.py:836: in wrapper 2025-10-10T02:02:57.0354250Z func(*args, **kwargs) 2025-10-10T02:02:57.0354431Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0354527Z LLM( 2025-10-10T02:02:57.0354855Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0356125Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0356504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0357455Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0357799Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:57.0359178Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:57.0359534Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:57.0360515Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:57.0360916Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:57.0362195Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:57.0362589Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:57.0363578Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:57.0363993Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:57.0365486Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:57.0365882Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:57.0366823Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:57.0367196Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:57.0368435Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:57.0368890Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:57.0369771Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:57.0370214Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:57.0371488Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:57.0371890Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:57.0372860Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:57.0373033Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0373074Z 2025-10-10T02:02:57.0374533Z > raise ValueError( 2025-10-10T02:02:57.0374965Z f"Model architectures {architectures} failed " 2025-10-10T02:02:57.0375306Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:57.0376000Z E ValueError: Model architectures ['PixtralForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.0376007Z 2025-10-10T02:02:57.0376393Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:57.0376653Z ______________ test_can_initialize_large_subset[JambaForCausalLM] ______________ 2025-10-10T02:02:57.0376659Z 2025-10-10T02:02:57.0381422Z model_arch = 'JambaForCausalLM' 2025-10-10T02:02:57.0381907Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa61f1d0> 2025-10-10T02:02:57.0381919Z 2025-10-10T02:02:57.0382622Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0382976Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0383194Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0383623Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0383736Z  2025-10-10T02:02:57.0384003Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0384111Z  test. 2025-10-10T02:02:57.0384225Z  """ 2025-10-10T02:02:57.0384526Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0385080Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0385339Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0385358Z 2025-10-10T02:02:57.0385511Z models/test_initialization.py:131: 2025-10-10T02:02:57.0385688Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0385908Z utils.py:892: in wrapper 2025-10-10T02:02:57.0386083Z raise original_exception 2025-10-10T02:02:57.0386195Z utils.py:836: in wrapper 2025-10-10T02:02:57.0386319Z func(*args, **kwargs) 2025-10-10T02:02:57.0386501Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0386612Z LLM( 2025-10-10T02:02:57.0386934Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0387747Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0388124Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0389045Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0389395Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0390430Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0390859Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0392513Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0393085Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0393337Z super().__init__( 2025-10-10T02:02:57.0393833Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0395303Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0395547Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0396591Z next(self.gen) 2025-10-10T02:02:57.0397163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0397341Z wait_for_engine_startup( 2025-10-10T02:02:57.0397578Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0397584Z 2025-10-10T02:02:57.0399537Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0399786Z "See root cause above. " 2025-10-10T02:02:57.0400199Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0400774Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0400785Z 2025-10-10T02:02:57.0401123Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0401387Z __________ test_can_initialize_large_subset[BailingMoeV2ForCausalLM] ___________ 2025-10-10T02:02:57.0401391Z 2025-10-10T02:02:57.0401487Z model_arch = 'BailingMoeV2ForCausalLM' 2025-10-10T02:02:57.0401660Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6457c0> 2025-10-10T02:02:57.0401666Z 2025-10-10T02:02:57.0406424Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0406958Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0407216Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0407604Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0407703Z  2025-10-10T02:02:57.0407968Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0408068Z  test. 2025-10-10T02:02:57.0408178Z  """ 2025-10-10T02:02:57.0408552Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0409115Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0409472Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0409478Z 2025-10-10T02:02:57.0409624Z models/test_initialization.py:131: 2025-10-10T02:02:57.0409725Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0409950Z utils.py:892: in wrapper 2025-10-10T02:02:57.0410282Z raise original_exception 2025-10-10T02:02:57.0410457Z utils.py:836: in wrapper 2025-10-10T02:02:57.0410683Z func(*args, **kwargs) 2025-10-10T02:02:57.0410961Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0411064Z LLM( 2025-10-10T02:02:57.0411385Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0411937Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0412347Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0412583Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0413122Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0413568Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0414256Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0414954Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0415414Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0415710Z super().__init__( 2025-10-10T02:02:57.0416342Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0417124Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0417648Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0417906Z next(self.gen) 2025-10-10T02:02:57.0418371Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0418546Z wait_for_engine_startup( 2025-10-10T02:02:57.0418679Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0418688Z 2025-10-10T02:02:57.0420847Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0421173Z "See root cause above. " 2025-10-10T02:02:57.0421622Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0422126Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0422135Z 2025-10-10T02:02:57.0422553Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0422856Z _____________ test_can_initialize_large_subset[TeleFLMForCausalLM] _____________ 2025-10-10T02:02:57.0422871Z 2025-10-10T02:02:57.0422976Z model_arch = 'TeleFLMForCausalLM' 2025-10-10T02:02:57.0423192Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212428dd0> 2025-10-10T02:02:57.0423197Z 2025-10-10T02:02:57.0426849Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0427300Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0427561Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0427794Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0427884Z  2025-10-10T02:02:57.0428140Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0428237Z  test. 2025-10-10T02:02:57.0428407Z  """ 2025-10-10T02:02:57.0428791Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0429216Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0429471Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0429476Z 2025-10-10T02:02:57.0429629Z models/test_initialization.py:131: 2025-10-10T02:02:57.0429741Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0429911Z utils.py:892: in wrapper 2025-10-10T02:02:57.0430088Z raise original_exception 2025-10-10T02:02:57.0430197Z utils.py:836: in wrapper 2025-10-10T02:02:57.0430363Z func(*args, **kwargs) 2025-10-10T02:02:57.0430607Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0430712Z LLM( 2025-10-10T02:02:57.0431042Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0431395Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0431781Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0432571Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0432935Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0434518Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0434944Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0435232Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0435788Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0436216Z super().__init__( 2025-10-10T02:02:57.0436598Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0437214Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0437570Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0437849Z next(self.gen) 2025-10-10T02:02:57.0438413Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0438572Z wait_for_engine_startup( 2025-10-10T02:02:57.0438674Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0438679Z 2025-10-10T02:02:57.0440620Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0441034Z "See root cause above. " 2025-10-10T02:02:57.0441576Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0441966Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0441971Z 2025-10-10T02:02:57.0442303Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0442599Z ________________ test_can_initialize_large_subset[GteNewModel] _________________ 2025-10-10T02:02:57.0442604Z 2025-10-10T02:02:57.0442683Z model_arch = 'GteNewModel' 2025-10-10T02:02:57.0442856Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6de750> 2025-10-10T02:02:57.0442860Z 2025-10-10T02:02:57.0447189Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0447597Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0447943Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0448259Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0448355Z  2025-10-10T02:02:57.0448623Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0448719Z  test. 2025-10-10T02:02:57.0448837Z  """ 2025-10-10T02:02:57.0449348Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0449927Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0450180Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0450188Z 2025-10-10T02:02:57.0450332Z models/test_initialization.py:131: 2025-10-10T02:02:57.0450434Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0450546Z utils.py:892: in wrapper 2025-10-10T02:02:57.0450710Z raise original_exception 2025-10-10T02:02:57.0450873Z utils.py:836: in wrapper 2025-10-10T02:02:57.0451096Z func(*args, **kwargs) 2025-10-10T02:02:57.0451353Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0451462Z LLM( 2025-10-10T02:02:57.0451804Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0452198Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0452604Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0452834Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0453326Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0453722Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0454085Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0455065Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0455578Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0456025Z super().__init__( 2025-10-10T02:02:57.0456560Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0457110Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0457438Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0457835Z next(self.gen) 2025-10-10T02:02:57.0458382Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0458531Z wait_for_engine_startup( 2025-10-10T02:02:57.0458633Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0458638Z 2025-10-10T02:02:57.0460655Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0461059Z "See root cause above. " 2025-10-10T02:02:57.0461432Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0461821Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0461829Z 2025-10-10T02:02:57.0462152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0462404Z _____________ test_can_initialize_large_subset[DotsOCRForCausalLM] _____________ 2025-10-10T02:02:57.0462408Z 2025-10-10T02:02:57.0462491Z model_arch = 'DotsOCRForCausalLM' 2025-10-10T02:02:57.0462685Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6cc710> 2025-10-10T02:02:57.0462691Z 2025-10-10T02:02:57.0467387Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0467834Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0468029Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0468262Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0468355Z  2025-10-10T02:02:57.0468611Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0468736Z  test. 2025-10-10T02:02:57.0468927Z  """ 2025-10-10T02:02:57.0469239Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0469661Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0469892Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0469896Z 2025-10-10T02:02:57.0470039Z models/test_initialization.py:131: 2025-10-10T02:02:57.0470145Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0470262Z utils.py:892: in wrapper 2025-10-10T02:02:57.0470431Z raise original_exception 2025-10-10T02:02:57.0470537Z utils.py:836: in wrapper 2025-10-10T02:02:57.0470676Z func(*args, **kwargs) 2025-10-10T02:02:57.0470937Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0471113Z LLM( 2025-10-10T02:02:57.0471481Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0471704Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0472171Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0472702Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0473219Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0473870Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0474224Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0475247Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0475735Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0475928Z super().__init__( 2025-10-10T02:02:57.0476267Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0477187Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0477523Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0478261Z next(self.gen) 2025-10-10T02:02:57.0478664Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0478815Z wait_for_engine_startup( 2025-10-10T02:02:57.0478925Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0478935Z 2025-10-10T02:02:57.0481281Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0481550Z "See root cause above. " 2025-10-10T02:02:57.0481916Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0482307Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0482312Z 2025-10-10T02:02:57.0482652Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0482934Z ______ test_can_initialize_large_subset[RobertaForSequenceClassification] ______ 2025-10-10T02:02:57.0482941Z 2025-10-10T02:02:57.0483058Z model_arch = 'RobertaForSequenceClassification' 2025-10-10T02:02:57.0483235Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6dcc50> 2025-10-10T02:02:57.0483239Z 2025-10-10T02:02:57.0487634Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0488089Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0488292Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0488526Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0488615Z  2025-10-10T02:02:57.0488962Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0489111Z  test. 2025-10-10T02:02:57.0489240Z  """ 2025-10-10T02:02:57.0489679Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0490147Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0490463Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0490468Z 2025-10-10T02:02:57.0490605Z models/test_initialization.py:131: 2025-10-10T02:02:57.0490712Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0490829Z utils.py:892: in wrapper 2025-10-10T02:02:57.0490995Z raise original_exception 2025-10-10T02:02:57.0491128Z utils.py:836: in wrapper 2025-10-10T02:02:57.0491345Z func(*args, **kwargs) 2025-10-10T02:02:57.0491651Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0491759Z LLM( 2025-10-10T02:02:57.0492082Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0492432Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0492811Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0493035Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0493358Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0494124Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0494478Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0495480Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0495967Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0496559Z super().__init__( 2025-10-10T02:02:57.0497120Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0497724Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0498158Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0498723Z next(self.gen) 2025-10-10T02:02:57.0499206Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0499383Z wait_for_engine_startup( 2025-10-10T02:02:57.0499509Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0499516Z 2025-10-10T02:02:57.0501412Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0501825Z "See root cause above. " 2025-10-10T02:02:57.0502275Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0502933Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0503019Z 2025-10-10T02:02:57.0503440Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0503754Z ______________ test_can_initialize_large_subset[BloomForCausalLM] ______________ 2025-10-10T02:02:57.0503758Z 2025-10-10T02:02:57.0503843Z model_arch = 'BloomForCausalLM' 2025-10-10T02:02:57.0504040Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6ceea0> 2025-10-10T02:02:57.0504110Z 2025-10-10T02:02:57.0507944Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0508352Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0508543Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0508774Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0508862Z  2025-10-10T02:02:57.0509220Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0509325Z  test. 2025-10-10T02:02:57.0509461Z  """ 2025-10-10T02:02:57.0509843Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0510280Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0510510Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0510515Z 2025-10-10T02:02:57.0510662Z models/test_initialization.py:131: 2025-10-10T02:02:57.0510767Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0510891Z utils.py:892: in wrapper 2025-10-10T02:02:57.0511066Z raise original_exception 2025-10-10T02:02:57.0511177Z utils.py:836: in wrapper 2025-10-10T02:02:57.0511297Z func(*args, **kwargs) 2025-10-10T02:02:57.0511571Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0511678Z LLM( 2025-10-10T02:02:57.0512005Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0512243Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0512685Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0513339Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0513748Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0514543Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0514914Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0515948Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0516300Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0516545Z super().__init__( 2025-10-10T02:02:57.0516863Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0517816Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0518206Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0518753Z next(self.gen) 2025-10-10T02:02:57.0519208Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0519344Z wait_for_engine_startup( 2025-10-10T02:02:57.0519446Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0519505Z 2025-10-10T02:02:57.0521453Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0521794Z "See root cause above. " 2025-10-10T02:02:57.0522164Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0522611Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0522615Z 2025-10-10T02:02:57.0522943Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0523209Z ________ test_can_initialize_large_subset[KeyeForConditionalGeneration] ________ 2025-10-10T02:02:57.0523215Z 2025-10-10T02:02:57.0523314Z model_arch = 'KeyeForConditionalGeneration' 2025-10-10T02:02:57.0523484Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6ddd60> 2025-10-10T02:02:57.0523488Z 2025-10-10T02:02:57.0528496Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0529110Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0529374Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0529674Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0529839Z  2025-10-10T02:02:57.0530291Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0530450Z  test. 2025-10-10T02:02:57.0530706Z  """ 2025-10-10T02:02:57.0531122Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0531557Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0531972Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0531986Z 2025-10-10T02:02:57.0532190Z models/test_initialization.py:131: 2025-10-10T02:02:57.0532318Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0532457Z utils.py:892: in wrapper 2025-10-10T02:02:57.0532664Z raise original_exception 2025-10-10T02:02:57.0532805Z utils.py:836: in wrapper 2025-10-10T02:02:57.0533081Z func(*args, **kwargs) 2025-10-10T02:02:57.0533378Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0533512Z LLM( 2025-10-10T02:02:57.0533872Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0534180Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0534676Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0534942Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0535273Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0535645Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0536040Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0536390Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0536855Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0537071Z super().__init__( 2025-10-10T02:02:57.0537411Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0538251Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0538490Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0539829Z next(self.gen) 2025-10-10T02:02:57.0540193Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0540327Z wait_for_engine_startup( 2025-10-10T02:02:57.0540430Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0540438Z 2025-10-10T02:02:57.0541785Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0542036Z "See root cause above. " 2025-10-10T02:02:57.0542456Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0542996Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0543002Z 2025-10-10T02:02:57.0543339Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0543606Z _____ test_can_initialize_large_subset[Qwen2_5_VLForConditionalGeneration] _____ 2025-10-10T02:02:57.0543610Z 2025-10-10T02:02:57.0543718Z model_arch = 'Qwen2_5_VLForConditionalGeneration' 2025-10-10T02:02:57.0543890Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6ce420> 2025-10-10T02:02:57.0543894Z 2025-10-10T02:02:57.0548481Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0548891Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0549135Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0549380Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0549473Z  2025-10-10T02:02:57.0549740Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0549844Z  test. 2025-10-10T02:02:57.0549985Z  """ 2025-10-10T02:02:57.0550392Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0550946Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0551254Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0551264Z 2025-10-10T02:02:57.0551407Z models/test_initialization.py:131: 2025-10-10T02:02:57.0551505Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0551692Z utils.py:892: in wrapper 2025-10-10T02:02:57.0551921Z raise original_exception 2025-10-10T02:02:57.0552117Z utils.py:836: in wrapper 2025-10-10T02:02:57.0552244Z func(*args, **kwargs) 2025-10-10T02:02:57.0552421Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0552523Z LLM( 2025-10-10T02:02:57.0552863Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0553197Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0553637Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0553883Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0554338Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0555818Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0556289Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0556758Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0557111Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0558366Z super().__init__( 2025-10-10T02:02:57.0558850Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0559789Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0560039Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0560481Z next(self.gen) 2025-10-10T02:02:57.0560871Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0561093Z wait_for_engine_startup( 2025-10-10T02:02:57.0561196Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0561201Z 2025-10-10T02:02:57.0563691Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0564096Z "See root cause above. " 2025-10-10T02:02:57.0564468Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0564862Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0564876Z 2025-10-10T02:02:57.0565208Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0565452Z ______________ test_can_initialize_large_subset[Qwen2ForCausalLM] ______________ 2025-10-10T02:02:57.0565455Z 2025-10-10T02:02:57.0565617Z model_arch = 'Qwen2ForCausalLM' 2025-10-10T02:02:57.0565845Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa4028d0> 2025-10-10T02:02:57.0565849Z 2025-10-10T02:02:57.0570891Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0571222Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0571477Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0571722Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0571878Z  2025-10-10T02:02:57.0572270Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0572442Z  test. 2025-10-10T02:02:57.0572564Z  """ 2025-10-10T02:02:57.0572878Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0573392Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0573657Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0573662Z 2025-10-10T02:02:57.0573828Z models/test_initialization.py:131: 2025-10-10T02:02:57.0574002Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0574151Z utils.py:892: in wrapper 2025-10-10T02:02:57.0574324Z raise original_exception 2025-10-10T02:02:57.0574435Z utils.py:836: in wrapper 2025-10-10T02:02:57.0574561Z func(*args, **kwargs) 2025-10-10T02:02:57.0574745Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0574848Z LLM( 2025-10-10T02:02:57.0575169Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0576141Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0576595Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0577338Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0577706Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0578539Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0578901Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0580012Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0580457Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0580786Z super().__init__( 2025-10-10T02:02:57.0581282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0582261Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0582599Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0583033Z next(self.gen) 2025-10-10T02:02:57.0583408Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0583611Z wait_for_engine_startup( 2025-10-10T02:02:57.0583835Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0583840Z 2025-10-10T02:02:57.0585856Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0586120Z "See root cause above. " 2025-10-10T02:02:57.0586665Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0587050Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0587055Z 2025-10-10T02:02:57.0587377Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0587650Z ______ test_can_initialize_large_subset[Idefics3ForConditionalGeneration] ______ 2025-10-10T02:02:57.0587654Z 2025-10-10T02:02:57.0587823Z model_arch = 'Idefics3ForConditionalGeneration' 2025-10-10T02:02:57.0587994Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212429430> 2025-10-10T02:02:57.0587998Z 2025-10-10T02:02:57.0593262Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0593716Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0593916Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0594143Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0594231Z  2025-10-10T02:02:57.0594494Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0594593Z  test. 2025-10-10T02:02:57.0594708Z  """ 2025-10-10T02:02:57.0594994Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0595453Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0595701Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0595706Z 2025-10-10T02:02:57.0595850Z models/test_initialization.py:131: 2025-10-10T02:02:57.0596020Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0596348Z utils.py:892: in wrapper 2025-10-10T02:02:57.0596525Z raise original_exception 2025-10-10T02:02:57.0596633Z utils.py:836: in wrapper 2025-10-10T02:02:57.0596750Z func(*args, **kwargs) 2025-10-10T02:02:57.0596936Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0597033Z LLM( 2025-10-10T02:02:57.0597363Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0598746Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0599351Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0599902Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0600247Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0601717Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0602156Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0603123Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0603622Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0604122Z super().__init__( 2025-10-10T02:02:57.0604837Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0606053Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0606438Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0606766Z next(self.gen) 2025-10-10T02:02:57.0607257Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0607608Z wait_for_engine_startup( 2025-10-10T02:02:57.0607716Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0607725Z 2025-10-10T02:02:57.0610147Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0610399Z "See root cause above. " 2025-10-10T02:02:57.0610763Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0611284Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0611291Z 2025-10-10T02:02:57.0611650Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0611902Z ______________ test_can_initialize_large_subset[Qwen3ForCausalLM] ______________ 2025-10-10T02:02:57.0611907Z 2025-10-10T02:02:57.0611988Z model_arch = 'Qwen3ForCausalLM' 2025-10-10T02:02:57.0612175Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa400ef0> 2025-10-10T02:02:57.0612179Z 2025-10-10T02:02:57.0617639Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0617964Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0618152Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0618388Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0618478Z  2025-10-10T02:02:57.0618747Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0618864Z  test. 2025-10-10T02:02:57.0618971Z  """ 2025-10-10T02:02:57.0619265Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0619603Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0619831Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0619835Z 2025-10-10T02:02:57.0619975Z models/test_initialization.py:131: 2025-10-10T02:02:57.0620142Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0620295Z utils.py:892: in wrapper 2025-10-10T02:02:57.0620478Z raise original_exception 2025-10-10T02:02:57.0620585Z utils.py:836: in wrapper 2025-10-10T02:02:57.0620708Z func(*args, **kwargs) 2025-10-10T02:02:57.0620883Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0621184Z LLM( 2025-10-10T02:02:57.0621526Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0623019Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0623530Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0624000Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0624351Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0625749Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0626183Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0627046Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0627530Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0628036Z super().__init__( 2025-10-10T02:02:57.0628379Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0629455Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0629758Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0630580Z next(self.gen) 2025-10-10T02:02:57.0631014Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0631179Z wait_for_engine_startup( 2025-10-10T02:02:57.0631282Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0631288Z 2025-10-10T02:02:57.0633792Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0634154Z "See root cause above. " 2025-10-10T02:02:57.0634689Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0635090Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0635094Z 2025-10-10T02:02:57.0635420Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0635674Z ______________ test_can_initialize_large_subset[Dots1ForCausalLM] ______________ 2025-10-10T02:02:57.0635678Z 2025-10-10T02:02:57.0635758Z model_arch = 'Dots1ForCausalLM' 2025-10-10T02:02:57.0635931Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124299a0> 2025-10-10T02:02:57.0635935Z 2025-10-10T02:02:57.0641200Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0641641Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0641896Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0642125Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0642216Z  2025-10-10T02:02:57.0642473Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0642615Z  test. 2025-10-10T02:02:57.0642725Z  """ 2025-10-10T02:02:57.0643015Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0643357Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0643577Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0643583Z 2025-10-10T02:02:57.0643723Z models/test_initialization.py:131: 2025-10-10T02:02:57.0643866Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0643985Z utils.py:892: in wrapper 2025-10-10T02:02:57.0644151Z raise original_exception 2025-10-10T02:02:57.0644254Z utils.py:836: in wrapper 2025-10-10T02:02:57.0644382Z func(*args, **kwargs) 2025-10-10T02:02:57.0644556Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0644658Z LLM( 2025-10-10T02:02:57.0644968Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0646211Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0646733Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0647179Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0647633Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0648564Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0649121Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0649825Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0650268Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0650674Z super().__init__( 2025-10-10T02:02:57.0651053Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0651763Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0652010Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0652650Z next(self.gen) 2025-10-10T02:02:57.0653049Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0653323Z wait_for_engine_startup( 2025-10-10T02:02:57.0653430Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0653434Z 2025-10-10T02:02:57.0655631Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0656019Z "See root cause above. " 2025-10-10T02:02:57.0656444Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0656835Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0656879Z 2025-10-10T02:02:57.0657199Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0657447Z ______________ test_can_initialize_large_subset[JAISLMHeadModel] _______________ 2025-10-10T02:02:57.0657451Z 2025-10-10T02:02:57.0657532Z model_arch = 'JAISLMHeadModel' 2025-10-10T02:02:57.0657702Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa400950> 2025-10-10T02:02:57.0657705Z 2025-10-10T02:02:57.0662543Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0662929Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0663139Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0663502Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0663605Z  2025-10-10T02:02:57.0663859Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0663957Z  test. 2025-10-10T02:02:57.0664068Z  """ 2025-10-10T02:02:57.0664358Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0664696Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0665051Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0665058Z 2025-10-10T02:02:57.0665208Z models/test_initialization.py:131: 2025-10-10T02:02:57.0665307Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0665420Z utils.py:892: in wrapper 2025-10-10T02:02:57.0665641Z raise original_exception 2025-10-10T02:02:57.0665818Z utils.py:836: in wrapper 2025-10-10T02:02:57.0665945Z func(*args, **kwargs) 2025-10-10T02:02:57.0666119Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0666461Z LLM( 2025-10-10T02:02:57.0666902Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0668327Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0668965Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0669292Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0669729Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0670901Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0671328Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0672509Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0673232Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0673534Z super().__init__( 2025-10-10T02:02:57.0673980Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0674800Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0675101Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0675795Z next(self.gen) 2025-10-10T02:02:57.0676267Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0676409Z wait_for_engine_startup( 2025-10-10T02:02:57.0676511Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0676522Z 2025-10-10T02:02:57.0678841Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0679271Z "See root cause above. " 2025-10-10T02:02:57.0679785Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0680177Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0680183Z 2025-10-10T02:02:57.0680507Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0680759Z ____________ test_can_initialize_large_subset[MiniCPM3ForCausalLM] _____________ 2025-10-10T02:02:57.0680765Z 2025-10-10T02:02:57.0680852Z model_arch = 'MiniCPM3ForCausalLM' 2025-10-10T02:02:57.0681029Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa278b60> 2025-10-10T02:02:57.0681036Z 2025-10-10T02:02:57.0686127Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0686455Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0686772Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0687016Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0687104Z  2025-10-10T02:02:57.0687364Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0687459Z  test. 2025-10-10T02:02:57.0687571Z  """ 2025-10-10T02:02:57.0687874Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0688228Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0688597Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0688606Z 2025-10-10T02:02:57.0688752Z models/test_initialization.py:131: 2025-10-10T02:02:57.0688856Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0689017Z utils.py:892: in wrapper 2025-10-10T02:02:57.0689268Z raise original_exception 2025-10-10T02:02:57.0689391Z utils.py:836: in wrapper 2025-10-10T02:02:57.0689511Z func(*args, **kwargs) 2025-10-10T02:02:57.0689764Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0689905Z LLM( 2025-10-10T02:02:57.0690230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0691211Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0691585Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0692446Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0692781Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0694129Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0694487Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0695850Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0699195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0699565Z super().__init__( 2025-10-10T02:02:57.0699927Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0700170Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0700400Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0700567Z next(self.gen) 2025-10-10T02:02:57.0700916Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0701060Z wait_for_engine_startup( 2025-10-10T02:02:57.0701241Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0701249Z 2025-10-10T02:02:57.0701989Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0702234Z "See root cause above. " 2025-10-10T02:02:57.0702733Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0703251Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0703257Z 2025-10-10T02:02:57.0703595Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0703852Z ________________ test_can_initialize_large_subset[MistralModel] ________________ 2025-10-10T02:02:57.0703858Z 2025-10-10T02:02:57.0703934Z model_arch = 'MistralModel' 2025-10-10T02:02:57.0704112Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6deb10> 2025-10-10T02:02:57.0704116Z 2025-10-10T02:02:57.0709240Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0709575Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0709813Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0710160Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0710253Z  2025-10-10T02:02:57.0710516Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0710727Z  test. 2025-10-10T02:02:57.0710839Z  """ 2025-10-10T02:02:57.0711141Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0711619Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0711965Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0711971Z 2025-10-10T02:02:57.0712214Z models/test_initialization.py:131: 2025-10-10T02:02:57.0712345Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0712466Z utils.py:892: in wrapper 2025-10-10T02:02:57.0712639Z raise original_exception 2025-10-10T02:02:57.0712755Z utils.py:836: in wrapper 2025-10-10T02:02:57.0712872Z func(*args, **kwargs) 2025-10-10T02:02:57.0713125Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0713308Z LLM( 2025-10-10T02:02:57.0713700Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0714276Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0714786Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0715377Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0715718Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0717100Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0717462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0718741Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0719323Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0719669Z super().__init__( 2025-10-10T02:02:57.0720035Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0721309Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0721552Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0723024Z next(self.gen) 2025-10-10T02:02:57.0723423Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0723558Z wait_for_engine_startup( 2025-10-10T02:02:57.0723673Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0723679Z 2025-10-10T02:02:57.0725399Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0725795Z "See root cause above. " 2025-10-10T02:02:57.0726188Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0726579Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0726659Z 2025-10-10T02:02:57.0726989Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0727250Z ____________ test_can_initialize_large_subset[DeepseekForCausalLM] _____________ 2025-10-10T02:02:57.0727255Z 2025-10-10T02:02:57.0727340Z model_arch = 'DeepseekForCausalLM' 2025-10-10T02:02:57.0727556Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa27a8a0> 2025-10-10T02:02:57.0727560Z 2025-10-10T02:02:57.0732343Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0732664Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0732899Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0733211Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0733375Z  2025-10-10T02:02:57.0733696Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0733806Z  test. 2025-10-10T02:02:57.0733923Z  """ 2025-10-10T02:02:57.0734338Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0734719Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0735061Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0735075Z 2025-10-10T02:02:57.0735228Z models/test_initialization.py:131: 2025-10-10T02:02:57.0735329Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0735446Z utils.py:892: in wrapper 2025-10-10T02:02:57.0735618Z raise original_exception 2025-10-10T02:02:57.0735725Z utils.py:836: in wrapper 2025-10-10T02:02:57.0735847Z func(*args, **kwargs) 2025-10-10T02:02:57.0736053Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0736238Z LLM( 2025-10-10T02:02:57.0736590Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0736828Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0737316Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0738007Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0738479Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0739197Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0739662Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0740538Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0741163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0741400Z super().__init__( 2025-10-10T02:02:57.0741851Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0742556Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0743774Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0744116Z next(self.gen) 2025-10-10T02:02:57.0744503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0744640Z wait_for_engine_startup( 2025-10-10T02:02:57.0744801Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0744806Z 2025-10-10T02:02:57.0746770Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0747014Z "See root cause above. " 2025-10-10T02:02:57.0747494Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0748022Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0748028Z 2025-10-10T02:02:57.0748373Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0748630Z _____________ test_can_initialize_large_subset[GPTNeoXForCausalLM] _____________ 2025-10-10T02:02:57.0748634Z 2025-10-10T02:02:57.0748718Z model_arch = 'GPTNeoXForCausalLM' 2025-10-10T02:02:57.0748895Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa403d40> 2025-10-10T02:02:57.0748899Z 2025-10-10T02:02:57.0753794Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0754124Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0754319Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0754557Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0754657Z  2025-10-10T02:02:57.0754917Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0755024Z  test. 2025-10-10T02:02:57.0755133Z  """ 2025-10-10T02:02:57.0755427Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0755762Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0755983Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0755990Z 2025-10-10T02:02:57.0756134Z models/test_initialization.py:131: 2025-10-10T02:02:57.0756237Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0756357Z utils.py:892: in wrapper 2025-10-10T02:02:57.0756523Z raise original_exception 2025-10-10T02:02:57.0756634Z utils.py:836: in wrapper 2025-10-10T02:02:57.0756758Z func(*args, **kwargs) 2025-10-10T02:02:57.0756934Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0757148Z LLM( 2025-10-10T02:02:57.0757491Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0758968Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0759762Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0760171Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0760554Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0761403Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0761951Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0762597Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0763041Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0763335Z super().__init__( 2025-10-10T02:02:57.0763691Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0764594Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0764921Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0765620Z next(self.gen) 2025-10-10T02:02:57.0766139Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0766285Z wait_for_engine_startup( 2025-10-10T02:02:57.0766391Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0766396Z 2025-10-10T02:02:57.0768916Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0769345Z "See root cause above. " 2025-10-10T02:02:57.0769871Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0770277Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0770285Z 2025-10-10T02:02:57.0770611Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0770892Z ______ test_can_initialize_large_subset[Qwen2VLForConditionalGeneration] _______ 2025-10-10T02:02:57.0770897Z 2025-10-10T02:02:57.0771006Z model_arch = 'Qwen2VLForConditionalGeneration' 2025-10-10T02:02:57.0771180Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa278830> 2025-10-10T02:02:57.0771183Z 2025-10-10T02:02:57.0775404Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0775839Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0776149Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0776400Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0776494Z  2025-10-10T02:02:57.0776753Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0776850Z  test. 2025-10-10T02:02:57.0777001Z  """ 2025-10-10T02:02:57.0777481Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0778126Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0778554Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0778562Z 2025-10-10T02:02:57.0778721Z models/test_initialization.py:131: 2025-10-10T02:02:57.0778821Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0779001Z utils.py:892: in wrapper 2025-10-10T02:02:57.0779189Z raise original_exception 2025-10-10T02:02:57.0779295Z utils.py:836: in wrapper 2025-10-10T02:02:57.0779418Z func(*args, **kwargs) 2025-10-10T02:02:57.0779724Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0779854Z LLM( 2025-10-10T02:02:57.0780192Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0780476Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0781050Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0781289Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0781630Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0782681Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0783037Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0784077Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0784546Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0784949Z super().__init__( 2025-10-10T02:02:57.0785325Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0786035Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0786281Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0786877Z next(self.gen) 2025-10-10T02:02:57.0787245Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0787471Z wait_for_engine_startup( 2025-10-10T02:02:57.0787575Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0787585Z 2025-10-10T02:02:57.0789728Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0790034Z "See root cause above. " 2025-10-10T02:02:57.0790492Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0790884Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0790888Z 2025-10-10T02:02:57.0791215Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0791478Z _____ test_can_initialize_large_subset[KeyeVL1_5ForConditionalGeneration] ______ 2025-10-10T02:02:57.0791548Z 2025-10-10T02:02:57.0791656Z model_arch = 'KeyeVL1_5ForConditionalGeneration' 2025-10-10T02:02:57.0791834Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6dd730> 2025-10-10T02:02:57.0791838Z 2025-10-10T02:02:57.0796682Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0797006Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0797302Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0797679Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0797782Z  2025-10-10T02:02:57.0798048Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0798156Z  test. 2025-10-10T02:02:57.0798359Z  """ 2025-10-10T02:02:57.0798794Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0799463Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0799824Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0799835Z 2025-10-10T02:02:57.0799991Z models/test_initialization.py:131: 2025-10-10T02:02:57.0800091Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0800213Z utils.py:892: in wrapper 2025-10-10T02:02:57.0800384Z raise original_exception 2025-10-10T02:02:57.0800498Z utils.py:836: in wrapper 2025-10-10T02:02:57.0800616Z func(*args, **kwargs) 2025-10-10T02:02:57.0800905Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0801032Z LLM( 2025-10-10T02:02:57.0801370Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0801593Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0802078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0802524Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0802958Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0803824Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0804171Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0805336Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0805684Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0806016Z super().__init__( 2025-10-10T02:02:57.0806383Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0807198Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0807462Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0808040Z next(self.gen) 2025-10-10T02:02:57.0808538Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0808801Z wait_for_engine_startup( 2025-10-10T02:02:57.0808913Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0808918Z 2025-10-10T02:02:57.0810741Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0811258Z "See root cause above. " 2025-10-10T02:02:57.0811671Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0812055Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0812067Z 2025-10-10T02:02:57.0812389Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0812704Z _____________ test_can_initialize_large_subset[MiniMaxForCausalLM] _____________ 2025-10-10T02:02:57.0812709Z 2025-10-10T02:02:57.0812846Z model_arch = 'MiniMaxForCausalLM' 2025-10-10T02:02:57.0813022Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa182c30> 2025-10-10T02:02:57.0813026Z 2025-10-10T02:02:57.0817454Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0817867Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0818059Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0818402Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0818509Z  2025-10-10T02:02:57.0818773Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0818873Z  test. 2025-10-10T02:02:57.0818994Z  """ 2025-10-10T02:02:57.0819286Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0819629Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0819897Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0819904Z 2025-10-10T02:02:57.0820114Z models/test_initialization.py:131: 2025-10-10T02:02:57.0820227Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0820343Z utils.py:892: in wrapper 2025-10-10T02:02:57.0820554Z raise original_exception 2025-10-10T02:02:57.0820745Z utils.py:836: in wrapper 2025-10-10T02:02:57.0820881Z func(*args, **kwargs) 2025-10-10T02:02:57.0821067Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0821166Z LLM( 2025-10-10T02:02:57.0821494Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0822850Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0823254Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0823858Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0824191Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0825538Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0825904Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0826960Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0827379Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0827831Z super().__init__( 2025-10-10T02:02:57.0828165Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0829533Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0829856Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0830246Z next(self.gen) 2025-10-10T02:02:57.0830786Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0831131Z wait_for_engine_startup( 2025-10-10T02:02:57.0831245Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0831254Z 2025-10-10T02:02:57.0833701Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0834012Z "See root cause above. " 2025-10-10T02:02:57.0834377Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0834759Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0834774Z 2025-10-10T02:02:57.0835121Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0835371Z ______________ test_can_initialize_large_subset[Olmo3ForCausalLM] ______________ 2025-10-10T02:02:57.0835375Z 2025-10-10T02:02:57.0835463Z model_arch = 'Olmo3ForCausalLM' 2025-10-10T02:02:57.0835637Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa278320> 2025-10-10T02:02:57.0835641Z 2025-10-10T02:02:57.0841040Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0841479Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0841682Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0841934Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0842035Z  2025-10-10T02:02:57.0842411Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0842518Z  test. 2025-10-10T02:02:57.0842635Z  """ 2025-10-10T02:02:57.0843001Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0843385Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0843621Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0843626Z 2025-10-10T02:02:57.0843869Z models/test_initialization.py:131: 2025-10-10T02:02:57.0844080Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0844214Z utils.py:892: in wrapper 2025-10-10T02:02:57.0844384Z raise original_exception 2025-10-10T02:02:57.0844497Z utils.py:836: in wrapper 2025-10-10T02:02:57.0844615Z func(*args, **kwargs) 2025-10-10T02:02:57.0844858Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0844967Z LLM( 2025-10-10T02:02:57.0845394Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0845923Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0846496Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0847050Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0847607Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0848305Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0848790Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0849583Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0850047Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0850383Z super().__init__( 2025-10-10T02:02:57.0850748Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0851485Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0851793Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0852176Z next(self.gen) 2025-10-10T02:02:57.0852658Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0852907Z wait_for_engine_startup( 2025-10-10T02:02:57.0853021Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0853029Z 2025-10-10T02:02:57.0855221Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0855575Z "See root cause above. " 2025-10-10T02:02:57.0855944Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0856440Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0856446Z 2025-10-10T02:02:57.0856782Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0857049Z ____________ test_can_initialize_large_subset[Qwen2MoeForCausalLM] _____________ 2025-10-10T02:02:57.0857053Z 2025-10-10T02:02:57.0857141Z model_arch = 'Qwen2MoeForCausalLM' 2025-10-10T02:02:57.0857340Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa1800e0> 2025-10-10T02:02:57.0857344Z 2025-10-10T02:02:57.0862215Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0862677Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0862946Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0863180Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0863273Z  2025-10-10T02:02:57.0863593Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0863698Z  test. 2025-10-10T02:02:57.0863808Z  """ 2025-10-10T02:02:57.0864230Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0864749Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0865088Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0865096Z 2025-10-10T02:02:57.0865425Z models/test_initialization.py:131: 2025-10-10T02:02:57.0865609Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0865764Z utils.py:892: in wrapper 2025-10-10T02:02:57.0865973Z raise original_exception 2025-10-10T02:02:57.0866103Z utils.py:836: in wrapper 2025-10-10T02:02:57.0866254Z func(*args, **kwargs) 2025-10-10T02:02:57.0866487Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0866711Z LLM( 2025-10-10T02:02:57.0867163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0867476Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0868156Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0868587Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0869144Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0870123Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0870703Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0871824Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0872351Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0872581Z super().__init__( 2025-10-10T02:02:57.0873142Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0874207Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0874540Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0874925Z next(self.gen) 2025-10-10T02:02:57.0875442Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0875590Z wait_for_engine_startup( 2025-10-10T02:02:57.0875692Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0875698Z 2025-10-10T02:02:57.0877889Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0878215Z "See root cause above. " 2025-10-10T02:02:57.0878718Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0879190Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0879253Z 2025-10-10T02:02:57.0879595Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0879882Z ___ test_can_initialize_large_subset[GraniteSpeechForConditionalGeneration] ____ 2025-10-10T02:02:57.0879886Z 2025-10-10T02:02:57.0880019Z model_arch = 'GraniteSpeechForConditionalGeneration' 2025-10-10T02:02:57.0880191Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa69bfb0> 2025-10-10T02:02:57.0880199Z 2025-10-10T02:02:57.0885438Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0885869Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0886185Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0886429Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0886517Z  2025-10-10T02:02:57.0886781Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0886878Z  test. 2025-10-10T02:02:57.0886999Z  """ 2025-10-10T02:02:57.0887296Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0887767Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0888065Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0888082Z 2025-10-10T02:02:57.0888302Z models/test_initialization.py:131: 2025-10-10T02:02:57.0888412Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0888533Z utils.py:892: in wrapper 2025-10-10T02:02:57.0888696Z raise original_exception 2025-10-10T02:02:57.0888807Z utils.py:836: in wrapper 2025-10-10T02:02:57.0888924Z func(*args, **kwargs) 2025-10-10T02:02:57.0889102Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0889206Z LLM( 2025-10-10T02:02:57.0889567Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0890518Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0891034Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0891530Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0891870Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0893239Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0893597Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0894672Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0895255Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0895584Z super().__init__( 2025-10-10T02:02:57.0895939Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0897088Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0897479Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0898237Z next(self.gen) 2025-10-10T02:02:57.0898706Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0898921Z wait_for_engine_startup( 2025-10-10T02:02:57.0899019Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0899030Z 2025-10-10T02:02:57.0901721Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0902001Z "See root cause above. " 2025-10-10T02:02:57.0902529Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0902924Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0902928Z 2025-10-10T02:02:57.0903251Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0903495Z _____________ test_can_initialize_large_subset[AquilaForCausalLM] ______________ 2025-10-10T02:02:57.0903502Z 2025-10-10T02:02:57.0903590Z model_arch = 'AquilaForCausalLM' 2025-10-10T02:02:57.0903766Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa181340> 2025-10-10T02:02:57.0903769Z 2025-10-10T02:02:57.0908672Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0908996Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0909175Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0909446Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0909603Z  2025-10-10T02:02:57.0909912Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0910015Z  test. 2025-10-10T02:02:57.0910124Z  """ 2025-10-10T02:02:57.0910422Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0910845Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0911138Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0911142Z 2025-10-10T02:02:57.0911277Z models/test_initialization.py:131: 2025-10-10T02:02:57.0911395Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0911592Z utils.py:892: in wrapper 2025-10-10T02:02:57.0911808Z raise original_exception 2025-10-10T02:02:57.0911923Z utils.py:836: in wrapper 2025-10-10T02:02:57.0912147Z func(*args, **kwargs) 2025-10-10T02:02:57.0912330Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0912432Z LLM( 2025-10-10T02:02:57.0912748Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0913633Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0914349Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0914925Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0915462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0916402Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0917036Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0917827Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0918405Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0918743Z super().__init__( 2025-10-10T02:02:57.0919409Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0919700Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0920123Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0921001Z next(self.gen) 2025-10-10T02:02:57.0921426Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0921575Z wait_for_engine_startup( 2025-10-10T02:02:57.0921676Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0921683Z 2025-10-10T02:02:57.0923897Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0924356Z "See root cause above. " 2025-10-10T02:02:57.0924821Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0925203Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0925208Z 2025-10-10T02:02:57.0925546Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0925787Z _____________ test_can_initialize_large_subset[Glm4MoeForCausalLM] _____________ 2025-10-10T02:02:57.0925794Z 2025-10-10T02:02:57.0925878Z model_arch = 'Glm4MoeForCausalLM' 2025-10-10T02:02:57.0926050Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa69aa50> 2025-10-10T02:02:57.0926056Z 2025-10-10T02:02:57.0930939Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0931265Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0931446Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0931719Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0931966Z  2025-10-10T02:02:57.0932245Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0932343Z  test. 2025-10-10T02:02:57.0932453Z  """ 2025-10-10T02:02:57.0932748Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0933281Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0933514Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0933523Z 2025-10-10T02:02:57.0933785Z models/test_initialization.py:131: 2025-10-10T02:02:57.0933919Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0934035Z utils.py:892: in wrapper 2025-10-10T02:02:57.0934205Z raise original_exception 2025-10-10T02:02:57.0934372Z utils.py:836: in wrapper 2025-10-10T02:02:57.0934495Z func(*args, **kwargs) 2025-10-10T02:02:57.0934726Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0934913Z LLM( 2025-10-10T02:02:57.0935294Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0935516Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0935982Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0936394Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0936727Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0938083Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0938438Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0939731Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0940206Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0940559Z super().__init__( 2025-10-10T02:02:57.0940916Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0942132Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0942386Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0943820Z next(self.gen) 2025-10-10T02:02:57.0944208Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0944343Z wait_for_engine_startup( 2025-10-10T02:02:57.0944448Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0944456Z 2025-10-10T02:02:57.0946541Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0946894Z "See root cause above. " 2025-10-10T02:02:57.0947266Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0947752Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0947758Z 2025-10-10T02:02:57.0948098Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0948350Z _____________ test_can_initialize_large_subset[PhiMoEForCausalLM] ______________ 2025-10-10T02:02:57.0948402Z 2025-10-10T02:02:57.0948485Z model_arch = 'PhiMoEForCausalLM' 2025-10-10T02:02:57.0948661Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa6de750> 2025-10-10T02:02:57.0948665Z 2025-10-10T02:02:57.0953302Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0953758Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0953963Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0954254Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0954402Z  2025-10-10T02:02:57.0954666Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0954767Z  test. 2025-10-10T02:02:57.0954974Z  """ 2025-10-10T02:02:57.0955312Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0955768Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0956020Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0956025Z 2025-10-10T02:02:57.0956173Z models/test_initialization.py:131: 2025-10-10T02:02:57.0956274Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0956387Z utils.py:892: in wrapper 2025-10-10T02:02:57.0956556Z raise original_exception 2025-10-10T02:02:57.0956661Z utils.py:836: in wrapper 2025-10-10T02:02:57.0956780Z func(*args, **kwargs) 2025-10-10T02:02:57.0956962Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0957061Z LLM( 2025-10-10T02:02:57.0957514Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0958262Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0958805Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0959541Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0959980Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0960910Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0961357Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0962262Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0962852Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0963185Z super().__init__( 2025-10-10T02:02:57.0963621Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0964364Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0964621Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0965193Z next(self.gen) 2025-10-10T02:02:57.0965564Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0965774Z wait_for_engine_startup( 2025-10-10T02:02:57.0965873Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0965877Z 2025-10-10T02:02:57.0968199Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0968562Z "See root cause above. " 2025-10-10T02:02:57.0969003Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0969450Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0969457Z 2025-10-10T02:02:57.0969792Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0970058Z _______ test_can_initialize_large_subset[GPT2ForSequenceClassification] ________ 2025-10-10T02:02:57.0970061Z 2025-10-10T02:02:57.0970162Z model_arch = 'GPT2ForSequenceClassification' 2025-10-10T02:02:57.0970336Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa699b20> 2025-10-10T02:02:57.0970339Z 2025-10-10T02:02:57.0974895Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0975343Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0975541Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0975769Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0975856Z  2025-10-10T02:02:57.0976111Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0976214Z  test. 2025-10-10T02:02:57.0976323Z  """ 2025-10-10T02:02:57.0976615Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0977086Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0977320Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0977329Z 2025-10-10T02:02:57.0977514Z models/test_initialization.py:131: 2025-10-10T02:02:57.0977675Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0977800Z utils.py:892: in wrapper 2025-10-10T02:02:57.0977971Z raise original_exception 2025-10-10T02:02:57.0978083Z utils.py:836: in wrapper 2025-10-10T02:02:57.0978204Z func(*args, **kwargs) 2025-10-10T02:02:57.0978384Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.0978487Z LLM( 2025-10-10T02:02:57.0978806Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.0980453Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.0980925Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.0981308Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.0981775Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.0982675Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.0983167Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.0984094Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.0984453Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.0984785Z super().__init__( 2025-10-10T02:02:57.0985226Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.0985938Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.0986248Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.0986803Z next(self.gen) 2025-10-10T02:02:57.0987288Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.0987435Z wait_for_engine_startup( 2025-10-10T02:02:57.0987534Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0987539Z 2025-10-10T02:02:57.0989514Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.0989758Z "See root cause above. " 2025-10-10T02:02:57.0990277Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.0990682Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.0990686Z 2025-10-10T02:02:57.0991006Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.0991255Z ______________ test_can_initialize_large_subset[ArceeForCausalLM] ______________ 2025-10-10T02:02:57.0991259Z 2025-10-10T02:02:57.0991338Z model_arch = 'ArceeForCausalLM' 2025-10-10T02:02:57.0991515Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa180710> 2025-10-10T02:02:57.0991521Z 2025-10-10T02:02:57.0996940Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.0997258Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.0997438Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.0997659Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.0997745Z  2025-10-10T02:02:57.0998010Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.0998113Z  test. 2025-10-10T02:02:57.0998221Z  """ 2025-10-10T02:02:57.0998521Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.0998984Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.0999334Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.0999340Z 2025-10-10T02:02:57.0999481Z models/test_initialization.py:131: 2025-10-10T02:02:57.0999651Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.0999768Z utils.py:892: in wrapper 2025-10-10T02:02:57.0999934Z raise original_exception 2025-10-10T02:02:57.1000043Z utils.py:836: in wrapper 2025-10-10T02:02:57.1000367Z func(*args, **kwargs) 2025-10-10T02:02:57.1000616Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1000792Z LLM( 2025-10-10T02:02:57.1001199Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1002795Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1003174Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1003859Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1004321Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1004947Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1005415Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1006377Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1006732Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1007041Z super().__init__( 2025-10-10T02:02:57.1007488Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1008378Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1008666Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1009118Z next(self.gen) 2025-10-10T02:02:57.1009476Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1009611Z wait_for_engine_startup( 2025-10-10T02:02:57.1009721Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1009726Z 2025-10-10T02:02:57.1012167Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1012502Z "See root cause above. " 2025-10-10T02:02:57.1012872Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1013255Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1013259Z 2025-10-10T02:02:57.1013589Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1013843Z _____________ test_can_initialize_large_subset[Zamba2ForCausalLM] ______________ 2025-10-10T02:02:57.1013943Z 2025-10-10T02:02:57.1014034Z model_arch = 'Zamba2ForCausalLM' 2025-10-10T02:02:57.1014222Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa69b110> 2025-10-10T02:02:57.1014228Z 2025-10-10T02:02:57.1019220Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1019727Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1019955Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1020241Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1020338Z  2025-10-10T02:02:57.1020598Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1020703Z  test. 2025-10-10T02:02:57.1020876Z  """ 2025-10-10T02:02:57.1021227Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1021678Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1021913Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1021917Z 2025-10-10T02:02:57.1022075Z models/test_initialization.py:131: 2025-10-10T02:02:57.1022239Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1022365Z utils.py:892: in wrapper 2025-10-10T02:02:57.1022532Z raise original_exception 2025-10-10T02:02:57.1022636Z utils.py:836: in wrapper 2025-10-10T02:02:57.1022770Z func(*args, **kwargs) 2025-10-10T02:02:57.1022953Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1023052Z LLM( 2025-10-10T02:02:57.1023374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1024669Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1025139Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1025671Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1026009Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1027463Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1027833Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1029217Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1029581Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1029760Z super().__init__( 2025-10-10T02:02:57.1030099Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1031588Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1031825Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1032162Z next(self.gen) 2025-10-10T02:02:57.1032601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1033046Z wait_for_engine_startup( 2025-10-10T02:02:57.1033154Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1033162Z 2025-10-10T02:02:57.1035688Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1036068Z "See root cause above. " 2025-10-10T02:02:57.1036432Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1036819Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1036827Z 2025-10-10T02:02:57.1037200Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1037516Z ______ test_can_initialize_large_subset[Step3VLForConditionalGeneration] _______ 2025-10-10T02:02:57.1037521Z 2025-10-10T02:02:57.1037632Z model_arch = 'Step3VLForConditionalGeneration' 2025-10-10T02:02:57.1037814Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa64e630> 2025-10-10T02:02:57.1037820Z 2025-10-10T02:02:57.1042948Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1043380Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1043568Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1043796Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1043898Z  2025-10-10T02:02:57.1044155Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1044259Z  test. 2025-10-10T02:02:57.1044369Z  """ 2025-10-10T02:02:57.1044725Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1045096Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1045413Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1045426Z 2025-10-10T02:02:57.1045698Z models/test_initialization.py:131: 2025-10-10T02:02:57.1045813Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1045943Z utils.py:892: in wrapper 2025-10-10T02:02:57.1046121Z raise original_exception 2025-10-10T02:02:57.1046242Z utils.py:836: in wrapper 2025-10-10T02:02:57.1046361Z func(*args, **kwargs) 2025-10-10T02:02:57.1046539Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1046646Z LLM( 2025-10-10T02:02:57.1047064Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1047574Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1047945Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1048714Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1049154Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1049928Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1050300Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1051199Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1051755Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1051941Z super().__init__( 2025-10-10T02:02:57.1052277Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1053242Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1053677Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1054294Z next(self.gen) 2025-10-10T02:02:57.1054734Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1054892Z wait_for_engine_startup( 2025-10-10T02:02:57.1054992Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1054999Z 2025-10-10T02:02:57.1056901Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1057292Z "See root cause above. " 2025-10-10T02:02:57.1057661Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1058048Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1058053Z 2025-10-10T02:02:57.1058374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1058627Z ___________ test_can_initialize_large_subset[GPTBigCodeForCausalLM] ____________ 2025-10-10T02:02:57.1058632Z 2025-10-10T02:02:57.1058720Z model_arch = 'GPTBigCodeForCausalLM' 2025-10-10T02:02:57.1058896Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa69aff0> 2025-10-10T02:02:57.1058899Z 2025-10-10T02:02:57.1063462Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1063934Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1064306Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1064558Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1064648Z  2025-10-10T02:02:57.1064905Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1065009Z  test. 2025-10-10T02:02:57.1065192Z  """ 2025-10-10T02:02:57.1065637Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1066086Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1066418Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1066501Z 2025-10-10T02:02:57.1066651Z models/test_initialization.py:131: 2025-10-10T02:02:57.1066752Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1066873Z utils.py:892: in wrapper 2025-10-10T02:02:57.1067043Z raise original_exception 2025-10-10T02:02:57.1067192Z utils.py:836: in wrapper 2025-10-10T02:02:57.1067491Z func(*args, **kwargs) 2025-10-10T02:02:57.1067735Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1067848Z LLM( 2025-10-10T02:02:57.1068174Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1076888Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1077372Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1077734Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1078478Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1078752Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1079232Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1079499Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1079839Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1080015Z super().__init__( 2025-10-10T02:02:57.1080338Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1080581Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1080809Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1080981Z next(self.gen) 2025-10-10T02:02:57.1081337Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1081471Z wait_for_engine_startup( 2025-10-10T02:02:57.1081570Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1081576Z 2025-10-10T02:02:57.1081984Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1082221Z "See root cause above. " 2025-10-10T02:02:57.1082577Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1082960Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1082966Z 2025-10-10T02:02:57.1083296Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1083550Z _______________ test_can_initialize_large_subset[GlmForCausalLM] _______________ 2025-10-10T02:02:57.1083554Z 2025-10-10T02:02:57.1083686Z model_arch = 'GlmForCausalLM' 2025-10-10T02:02:57.1083938Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa64d490> 2025-10-10T02:02:57.1083944Z 2025-10-10T02:02:57.1084947Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1085499Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1085857Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1086148Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1086330Z  2025-10-10T02:02:57.1086603Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1086703Z  test. 2025-10-10T02:02:57.1086901Z  """ 2025-10-10T02:02:57.1087452Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1088049Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1088367Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1088373Z 2025-10-10T02:02:57.1088569Z models/test_initialization.py:131: 2025-10-10T02:02:57.1088671Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1088851Z utils.py:892: in wrapper 2025-10-10T02:02:57.1089161Z raise original_exception 2025-10-10T02:02:57.1089293Z utils.py:836: in wrapper 2025-10-10T02:02:57.1089425Z func(*args, **kwargs) 2025-10-10T02:02:57.1089603Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1089701Z LLM( 2025-10-10T02:02:57.1090025Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1090249Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1090802Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1091073Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1091417Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1092461Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1092821Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1094239Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1094689Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1095042Z super().__init__( 2025-10-10T02:02:57.1095516Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1096657Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1097079Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1097860Z next(self.gen) 2025-10-10T02:02:57.1098272Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1098413Z wait_for_engine_startup( 2025-10-10T02:02:57.1098513Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1098518Z 2025-10-10T02:02:57.1100836Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1101359Z "See root cause above. " 2025-10-10T02:02:57.1101727Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1102207Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1102212Z 2025-10-10T02:02:57.1102541Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1102803Z ___________ test_can_initialize_large_subset[Starcoder2ForCausalLM] ____________ 2025-10-10T02:02:57.1102808Z 2025-10-10T02:02:57.1102892Z model_arch = 'Starcoder2ForCausalLM' 2025-10-10T02:02:57.1103074Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa69ac90> 2025-10-10T02:02:57.1103077Z 2025-10-10T02:02:57.1107660Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1108146Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1108348Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1108571Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1108669Z  2025-10-10T02:02:57.1108929Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1109058Z  test. 2025-10-10T02:02:57.1109254Z  """ 2025-10-10T02:02:57.1109574Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1110051Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1110280Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1110288Z 2025-10-10T02:02:57.1110440Z models/test_initialization.py:131: 2025-10-10T02:02:57.1110537Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1110652Z utils.py:892: in wrapper 2025-10-10T02:02:57.1110816Z raise original_exception 2025-10-10T02:02:57.1110958Z utils.py:836: in wrapper 2025-10-10T02:02:57.1111173Z func(*args, **kwargs) 2025-10-10T02:02:57.1111356Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1111462Z LLM( 2025-10-10T02:02:57.1111786Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1112049Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1112505Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1112758Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1113087Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1114070Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1114428Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1115475Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1115893Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1116235Z super().__init__( 2025-10-10T02:02:57.1116675Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1117504Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1117794Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1118376Z next(self.gen) 2025-10-10T02:02:57.1118782Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1118938Z wait_for_engine_startup( 2025-10-10T02:02:57.1119113Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1119184Z 2025-10-10T02:02:57.1121283Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1121650Z "See root cause above. " 2025-10-10T02:02:57.1122030Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1122415Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1122427Z 2025-10-10T02:02:57.1122762Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1123006Z _____________ test_can_initialize_large_subset[GraniteForCausalLM] _____________ 2025-10-10T02:02:57.1123010Z 2025-10-10T02:02:57.1123096Z model_arch = 'GraniteForCausalLM' 2025-10-10T02:02:57.1123272Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221242aea0> 2025-10-10T02:02:57.1123275Z 2025-10-10T02:02:57.1127692Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1128019Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1128225Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1128620Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1128716Z  2025-10-10T02:02:57.1128979Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1129080Z  test. 2025-10-10T02:02:57.1129191Z  """ 2025-10-10T02:02:57.1129484Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1129900Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1130269Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1130274Z 2025-10-10T02:02:57.1130418Z models/test_initialization.py:131: 2025-10-10T02:02:57.1130538Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1130740Z utils.py:892: in wrapper 2025-10-10T02:02:57.1130917Z raise original_exception 2025-10-10T02:02:57.1131104Z utils.py:836: in wrapper 2025-10-10T02:02:57.1131220Z func(*args, **kwargs) 2025-10-10T02:02:57.1131394Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1131500Z LLM( 2025-10-10T02:02:57.1131828Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1132858Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1133226Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1134200Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1134536Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1135701Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1136106Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1137606Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1138055Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1138563Z super().__init__( 2025-10-10T02:02:57.1139016Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1140043Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1140350Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1140968Z next(self.gen) 2025-10-10T02:02:57.1141631Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1141835Z wait_for_engine_startup( 2025-10-10T02:02:57.1141972Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1141979Z 2025-10-10T02:02:57.1144277Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1144688Z "See root cause above. " 2025-10-10T02:02:57.1145268Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1145666Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1145677Z 2025-10-10T02:02:57.1146013Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1146297Z ______ test_can_initialize_large_subset[Tarsier2ForConditionalGeneration] ______ 2025-10-10T02:02:57.1146302Z 2025-10-10T02:02:57.1146420Z model_arch = 'Tarsier2ForConditionalGeneration' 2025-10-10T02:02:57.1146596Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa64cc50> 2025-10-10T02:02:57.1146600Z 2025-10-10T02:02:57.1151208Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1151635Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1151896Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1152226Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1152337Z  2025-10-10T02:02:57.1152609Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1152716Z  test. 2025-10-10T02:02:57.1152827Z  """ 2025-10-10T02:02:57.1153270Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1153668Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1153994Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1154001Z 2025-10-10T02:02:57.1154160Z models/test_initialization.py:131: 2025-10-10T02:02:57.1154262Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1154381Z utils.py:892: in wrapper 2025-10-10T02:02:57.1154618Z raise original_exception 2025-10-10T02:02:57.1154764Z utils.py:836: in wrapper 2025-10-10T02:02:57.1154890Z func(*args, **kwargs) 2025-10-10T02:02:57.1155061Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1155166Z LLM( 2025-10-10T02:02:57.1155481Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1156797Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1157171Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1157693Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1158037Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1159527Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1159879Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1161200Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1161550Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1161866Z super().__init__( 2025-10-10T02:02:57.1162195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1163616Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1163865Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1164440Z next(self.gen) 2025-10-10T02:02:57.1164955Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1165136Z wait_for_engine_startup( 2025-10-10T02:02:57.1165247Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1165252Z 2025-10-10T02:02:57.1167796Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1168041Z "See root cause above. " 2025-10-10T02:02:57.1168643Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1169136Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1169146Z 2025-10-10T02:02:57.1169486Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1169804Z ____ test_can_initialize_large_subset[XLMRobertaForSequenceClassification] _____ 2025-10-10T02:02:57.1169808Z 2025-10-10T02:02:57.1169930Z model_arch = 'XLMRobertaForSequenceClassification' 2025-10-10T02:02:57.1170101Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71c140> 2025-10-10T02:02:57.1170105Z 2025-10-10T02:02:57.1174915Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1175246Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1175521Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1175955Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1176065Z  2025-10-10T02:02:57.1176334Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1176438Z  test. 2025-10-10T02:02:57.1176546Z  """ 2025-10-10T02:02:57.1176844Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1177181Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1177529Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1177543Z 2025-10-10T02:02:57.1177696Z models/test_initialization.py:131: 2025-10-10T02:02:57.1177802Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1177914Z utils.py:892: in wrapper 2025-10-10T02:02:57.1178148Z raise original_exception 2025-10-10T02:02:57.1178334Z utils.py:836: in wrapper 2025-10-10T02:02:57.1178457Z func(*args, **kwargs) 2025-10-10T02:02:57.1178636Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1178734Z LLM( 2025-10-10T02:02:57.1179054Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1180394Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1180813Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1181385Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1181726Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1183686Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1184065Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1184334Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1184810Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1185150Z super().__init__( 2025-10-10T02:02:57.1185605Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1186590Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1186917Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1187482Z next(self.gen) 2025-10-10T02:02:57.1188066Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1188206Z wait_for_engine_startup( 2025-10-10T02:02:57.1188310Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1188315Z 2025-10-10T02:02:57.1190087Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1190399Z "See root cause above. " 2025-10-10T02:02:57.1190921Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1191325Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1191332Z 2025-10-10T02:02:57.1191657Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1191926Z ______ test_can_initialize_large_subset[GteNewForSequenceClassification] _______ 2025-10-10T02:02:57.1191930Z 2025-10-10T02:02:57.1192038Z model_arch = 'GteNewForSequenceClassification' 2025-10-10T02:02:57.1192211Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa698dd0> 2025-10-10T02:02:57.1192217Z 2025-10-10T02:02:57.1197044Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1197375Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1197563Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1197799Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1197953Z  2025-10-10T02:02:57.1198383Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1198488Z  test. 2025-10-10T02:02:57.1198602Z  """ 2025-10-10T02:02:57.1198897Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1199363Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1199590Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1199595Z 2025-10-10T02:02:57.1199836Z models/test_initialization.py:131: 2025-10-10T02:02:57.1199952Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1200080Z utils.py:892: in wrapper 2025-10-10T02:02:57.1200253Z raise original_exception 2025-10-10T02:02:57.1200408Z utils.py:836: in wrapper 2025-10-10T02:02:57.1200584Z func(*args, **kwargs) 2025-10-10T02:02:57.1200758Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1200858Z LLM( 2025-10-10T02:02:57.1201174Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1202306Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1202692Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1203408Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1203872Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1204632Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1204990Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1205982Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1206489Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1206838Z super().__init__( 2025-10-10T02:02:57.1207359Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1208392Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1208666Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1208879Z next(self.gen) 2025-10-10T02:02:57.1209300Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1209458Z wait_for_engine_startup( 2025-10-10T02:02:57.1209557Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1209566Z 2025-10-10T02:02:57.1211638Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1211927Z "See root cause above. " 2025-10-10T02:02:57.1212473Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1212881Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1212886Z 2025-10-10T02:02:57.1213214Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1213474Z ____________ test_can_initialize_large_subset[BaiChuanForCausalLM] _____________ 2025-10-10T02:02:57.1213481Z 2025-10-10T02:02:57.1213566Z model_arch = 'BaiChuanForCausalLM' 2025-10-10T02:02:57.1213744Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71e630> 2025-10-10T02:02:57.1213748Z 2025-10-10T02:02:57.1218711Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1219294Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1219557Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1219784Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1219876Z  2025-10-10T02:02:57.1220147Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1220333Z  test. 2025-10-10T02:02:57.1220655Z  """ 2025-10-10T02:02:57.1221167Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1221536Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1221756Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1221822Z 2025-10-10T02:02:57.1221965Z models/test_initialization.py:131: 2025-10-10T02:02:57.1222062Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1222184Z utils.py:892: in wrapper 2025-10-10T02:02:57.1222440Z raise original_exception 2025-10-10T02:02:57.1222587Z utils.py:836: in wrapper 2025-10-10T02:02:57.1222711Z func(*args, **kwargs) 2025-10-10T02:02:57.1222892Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1222993Z LLM( 2025-10-10T02:02:57.1223436Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1223824Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1224221Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1224576Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1225020Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1225257Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1225600Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1226576Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1227036Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1227223Z super().__init__( 2025-10-10T02:02:57.1227554Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1228484Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1228725Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1229423Z next(self.gen) 2025-10-10T02:02:57.1229841Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1230003Z wait_for_engine_startup( 2025-10-10T02:02:57.1230111Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1230117Z 2025-10-10T02:02:57.1232341Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1232639Z "See root cause above. " 2025-10-10T02:02:57.1233010Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1233403Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1233408Z 2025-10-10T02:02:57.1233731Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1234119Z ______ test_can_initialize_large_subset[Mistral3ForConditionalGeneration] ______ 2025-10-10T02:02:57.1234124Z 2025-10-10T02:02:57.1234242Z model_arch = 'Mistral3ForConditionalGeneration' 2025-10-10T02:02:57.1234416Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212429d30> 2025-10-10T02:02:57.1234420Z 2025-10-10T02:02:57.1238907Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1239447Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1239645Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1239885Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1239980Z  2025-10-10T02:02:57.1240333Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1240534Z  test. 2025-10-10T02:02:57.1240727Z  """ 2025-10-10T02:02:57.1241103Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1241525Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1241767Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1241772Z 2025-10-10T02:02:57.1241919Z models/test_initialization.py:131: 2025-10-10T02:02:57.1242027Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1242156Z utils.py:892: in wrapper 2025-10-10T02:02:57.1242418Z raise original_exception 2025-10-10T02:02:57.1242557Z utils.py:836: in wrapper 2025-10-10T02:02:57.1242681Z func(*args, **kwargs) 2025-10-10T02:02:57.1242874Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1242977Z LLM( 2025-10-10T02:02:57.1243376Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1243654Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1244037Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1244471Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1244924Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:57.1245940Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:57.1246307Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:57.1246898Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:57.1247408Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:57.1248286Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:57.1248747Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:57.1249476Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:57.1250081Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:57.1251087Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:57.1251563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:57.1252190Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:57.1252672Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:57.1253579Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:57.1254133Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:57.1254748Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:57.1255481Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:57.1256628Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:57.1257091Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:57.1257530Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:57.1257632Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1257642Z 2025-10-10T02:02:57.1259157Z > raise ValueError( 2025-10-10T02:02:57.1259766Z f"Model architectures {architectures} failed " 2025-10-10T02:02:57.1260318Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:57.1260801Z E ValueError: Model architectures ['Mistral3ForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.1260806Z 2025-10-10T02:02:57.1261190Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:57.1261449Z ____________ test_can_initialize_large_subset[LongCatFlashMTPModel] ____________ 2025-10-10T02:02:57.1261453Z 2025-10-10T02:02:57.1261543Z model_arch = 'LongCatFlashMTPModel' 2025-10-10T02:02:57.1261714Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71d010> 2025-10-10T02:02:57.1261718Z 2025-10-10T02:02:57.1266166Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1266521Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1266840Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1267150Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1267253Z  2025-10-10T02:02:57.1267528Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1267627Z  test. 2025-10-10T02:02:57.1267751Z  """ 2025-10-10T02:02:57.1268187Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1268695Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1269076Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1269081Z 2025-10-10T02:02:57.1269238Z models/test_initialization.py:131: 2025-10-10T02:02:57.1269343Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1269468Z utils.py:892: in wrapper 2025-10-10T02:02:57.1269698Z raise original_exception 2025-10-10T02:02:57.1269883Z utils.py:836: in wrapper 2025-10-10T02:02:57.1270111Z func(*args, **kwargs) 2025-10-10T02:02:57.1270336Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1270443Z LLM( 2025-10-10T02:02:57.1270774Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1271032Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1271620Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1271922Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1272364Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1272855Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1273207Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1274181Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1274668Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1275050Z super().__init__( 2025-10-10T02:02:57.1275501Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1276054Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1276356Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1276828Z next(self.gen) 2025-10-10T02:02:57.1277345Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1277499Z wait_for_engine_startup( 2025-10-10T02:02:57.1277609Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1277615Z 2025-10-10T02:02:57.1279650Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1279913Z "See root cause above. " 2025-10-10T02:02:57.1280443Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1280850Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1280856Z 2025-10-10T02:02:57.1281185Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1281461Z ______ test_can_initialize_large_subset[InternVLForConditionalGeneration] ______ 2025-10-10T02:02:57.1281466Z 2025-10-10T02:02:57.1281589Z model_arch = 'InternVLForConditionalGeneration' 2025-10-10T02:02:57.1281866Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa699460> 2025-10-10T02:02:57.1281870Z 2025-10-10T02:02:57.1286511Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1286985Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1287303Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1287542Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1287638Z  2025-10-10T02:02:57.1288028Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1288223Z  test. 2025-10-10T02:02:57.1288364Z  """ 2025-10-10T02:02:57.1288891Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1289373Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1289613Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1289619Z 2025-10-10T02:02:57.1289864Z models/test_initialization.py:131: 2025-10-10T02:02:57.1290044Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1290225Z utils.py:892: in wrapper 2025-10-10T02:02:57.1290408Z raise original_exception 2025-10-10T02:02:57.1290521Z utils.py:836: in wrapper 2025-10-10T02:02:57.1290649Z func(*args, **kwargs) 2025-10-10T02:02:57.1290895Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1291090Z LLM( 2025-10-10T02:02:57.1291498Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1291725Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1292158Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1292543Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1292888Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1293144Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1293594Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1294675Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1295059Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1295258Z super().__init__( 2025-10-10T02:02:57.1295764Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1296647Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1297053Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1297680Z next(self.gen) 2025-10-10T02:02:57.1298168Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1298479Z wait_for_engine_startup( 2025-10-10T02:02:57.1298718Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1298723Z 2025-10-10T02:02:57.1300782Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1301094Z "See root cause above. " 2025-10-10T02:02:57.1301659Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1302170Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1302177Z 2025-10-10T02:02:57.1302535Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1302804Z ______________ test_can_initialize_large_subset[Phi3ForCausalLM] _______________ 2025-10-10T02:02:57.1302810Z 2025-10-10T02:02:57.1302977Z model_arch = 'Phi3ForCausalLM' 2025-10-10T02:02:57.1303214Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71cd10> 2025-10-10T02:02:57.1303219Z 2025-10-10T02:02:57.1307743Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1308095Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1308289Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1308525Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1308635Z  2025-10-10T02:02:57.1309027Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1309144Z  test. 2025-10-10T02:02:57.1309261Z  """ 2025-10-10T02:02:57.1309715Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1310071Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1310294Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1310299Z 2025-10-10T02:02:57.1310441Z models/test_initialization.py:131: 2025-10-10T02:02:57.1310540Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1310712Z utils.py:892: in wrapper 2025-10-10T02:02:57.1310968Z raise original_exception 2025-10-10T02:02:57.1311095Z utils.py:836: in wrapper 2025-10-10T02:02:57.1311228Z func(*args, **kwargs) 2025-10-10T02:02:57.1311412Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1311527Z LLM( 2025-10-10T02:02:57.1311992Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1312226Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1312599Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1313009Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1313466Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1314196Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1314822Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1315782Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1316144Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1316406Z super().__init__( 2025-10-10T02:02:57.1316892Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1317822Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1318098Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1318685Z next(self.gen) 2025-10-10T02:02:57.1319324Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1319593Z wait_for_engine_startup( 2025-10-10T02:02:57.1319781Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1319788Z 2025-10-10T02:02:57.1321877Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1322200Z "See root cause above. " 2025-10-10T02:02:57.1322644Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1323124Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1323134Z 2025-10-10T02:02:57.1323549Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1323860Z ________________ test_can_initialize_large_subset[RobertaModel] ________________ 2025-10-10T02:02:57.1323865Z 2025-10-10T02:02:57.1323941Z model_arch = 'RobertaModel' 2025-10-10T02:02:57.1324114Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa711c40> 2025-10-10T02:02:57.1324120Z 2025-10-10T02:02:57.1328038Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1328356Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1328547Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1328773Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1328873Z  2025-10-10T02:02:57.1329135Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1329236Z  test. 2025-10-10T02:02:57.1329355Z  """ 2025-10-10T02:02:57.1329649Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1330001Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1330223Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1330227Z 2025-10-10T02:02:57.1330370Z models/test_initialization.py:131: 2025-10-10T02:02:57.1330473Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1330671Z utils.py:892: in wrapper 2025-10-10T02:02:57.1330843Z raise original_exception 2025-10-10T02:02:57.1330948Z utils.py:836: in wrapper 2025-10-10T02:02:57.1331092Z func(*args, **kwargs) 2025-10-10T02:02:57.1331280Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1331504Z LLM( 2025-10-10T02:02:57.1331838Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1332529Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1332900Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1333660Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1334046Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1334868Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1335269Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1336236Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1336587Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1337012Z super().__init__( 2025-10-10T02:02:57.1337561Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1338375Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1338698Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1339079Z next(self.gen) 2025-10-10T02:02:57.1339717Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1339888Z wait_for_engine_startup( 2025-10-10T02:02:57.1339997Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1340003Z 2025-10-10T02:02:57.1342003Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1342419Z "See root cause above. " 2025-10-10T02:02:57.1342858Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1343270Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1343276Z 2025-10-10T02:02:57.1343613Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1343881Z ___________ test_can_initialize_large_subset[DeepseekV2ForCausalLM] ____________ 2025-10-10T02:02:57.1343888Z 2025-10-10T02:02:57.1343974Z model_arch = 'DeepseekV2ForCausalLM' 2025-10-10T02:02:57.1344144Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221242a6c0> 2025-10-10T02:02:57.1344148Z 2025-10-10T02:02:57.1348387Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1348745Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1349032Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1349274Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1349368Z  2025-10-10T02:02:57.1349626Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1349778Z  test. 2025-10-10T02:02:57.1349905Z  """ 2025-10-10T02:02:57.1350199Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1350567Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1350897Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1350906Z 2025-10-10T02:02:57.1351051Z models/test_initialization.py:131: 2025-10-10T02:02:57.1351211Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1351370Z utils.py:892: in wrapper 2025-10-10T02:02:57.1351545Z raise original_exception 2025-10-10T02:02:57.1351651Z utils.py:836: in wrapper 2025-10-10T02:02:57.1351822Z func(*args, **kwargs) 2025-10-10T02:02:57.1352070Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1352171Z LLM( 2025-10-10T02:02:57.1352502Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1352798Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1353297Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1353720Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1354254Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1355057Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1355609Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1356530Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1356901Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1357237Z super().__init__( 2025-10-10T02:02:57.1357726Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1358434Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1358714Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1359290Z next(self.gen) 2025-10-10T02:02:57.1359713Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1359986Z wait_for_engine_startup( 2025-10-10T02:02:57.1360093Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1360098Z 2025-10-10T02:02:57.1362142Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1362477Z "See root cause above. " 2025-10-10T02:02:57.1363036Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1363456Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1363562Z 2025-10-10T02:02:57.1364066Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1364360Z ______________ test_can_initialize_large_subset[OrionForCausalLM] ______________ 2025-10-10T02:02:57.1364365Z 2025-10-10T02:02:57.1364460Z model_arch = 'OrionForCausalLM' 2025-10-10T02:02:57.1364666Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71e630> 2025-10-10T02:02:57.1364677Z 2025-10-10T02:02:57.1368946Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1369449Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1369691Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1370023Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1370136Z  2025-10-10T02:02:57.1370395Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1370489Z  test. 2025-10-10T02:02:57.1370607Z  """ 2025-10-10T02:02:57.1370896Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1371353Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1371587Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1371592Z 2025-10-10T02:02:57.1371747Z models/test_initialization.py:131: 2025-10-10T02:02:57.1371920Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1372063Z utils.py:892: in wrapper 2025-10-10T02:02:57.1372231Z raise original_exception 2025-10-10T02:02:57.1372340Z utils.py:836: in wrapper 2025-10-10T02:02:57.1372475Z func(*args, **kwargs) 2025-10-10T02:02:57.1372656Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1372751Z LLM( 2025-10-10T02:02:57.1373175Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1373403Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1373917Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1374330Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1374708Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1375673Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1376064Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1376993Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1377374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1377688Z super().__init__( 2025-10-10T02:02:57.1378020Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1378952Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1379219Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1379940Z next(self.gen) 2025-10-10T02:02:57.1380374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1380536Z wait_for_engine_startup( 2025-10-10T02:02:57.1380769Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1380777Z 2025-10-10T02:02:57.1382882Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1383355Z "See root cause above. " 2025-10-10T02:02:57.1383843Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1384303Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1384308Z 2025-10-10T02:02:57.1384635Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1384876Z ______________ test_can_initialize_large_subset[ModernBertModel] _______________ 2025-10-10T02:02:57.1384880Z 2025-10-10T02:02:57.1384959Z model_arch = 'ModernBertModel' 2025-10-10T02:02:57.1385139Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa710ef0> 2025-10-10T02:02:57.1385149Z 2025-10-10T02:02:57.1389227Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1389581Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1389926Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1390202Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1390296Z  2025-10-10T02:02:57.1390647Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1390754Z  test. 2025-10-10T02:02:57.1390868Z  """ 2025-10-10T02:02:57.1391300Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1391666Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1391890Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1391897Z 2025-10-10T02:02:57.1392050Z models/test_initialization.py:131: 2025-10-10T02:02:57.1392153Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1392273Z utils.py:892: in wrapper 2025-10-10T02:02:57.1392453Z raise original_exception 2025-10-10T02:02:57.1392560Z utils.py:836: in wrapper 2025-10-10T02:02:57.1392688Z func(*args, **kwargs) 2025-10-10T02:02:57.1392866Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1393049Z LLM( 2025-10-10T02:02:57.1393401Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1393706Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1394090Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1394560Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1394992Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1395779Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1396438Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1397302Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1397849Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1398141Z super().__init__( 2025-10-10T02:02:57.1398581Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1399479Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1399718Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1400341Z next(self.gen) 2025-10-10T02:02:57.1400712Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1400978Z wait_for_engine_startup( 2025-10-10T02:02:57.1401090Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1401102Z 2025-10-10T02:02:57.1403236Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1403498Z "See root cause above. " 2025-10-10T02:02:57.1403867Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1404252Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1404256Z 2025-10-10T02:02:57.1404588Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1404844Z ______________ test_can_initialize_large_subset[MiMoForCausalLM] _______________ 2025-10-10T02:02:57.1404850Z 2025-10-10T02:02:57.1404934Z model_arch = 'MiMoForCausalLM' 2025-10-10T02:02:57.1405123Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221242b020> 2025-10-10T02:02:57.1405127Z 2025-10-10T02:02:57.1409520Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1409917Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1410136Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1410369Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1410457Z  2025-10-10T02:02:57.1410710Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1410947Z  test. 2025-10-10T02:02:57.1411066Z  """ 2025-10-10T02:02:57.1411474Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1412023Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1412399Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1412404Z 2025-10-10T02:02:57.1412544Z models/test_initialization.py:131: 2025-10-10T02:02:57.1412642Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1412757Z utils.py:892: in wrapper 2025-10-10T02:02:57.1412920Z raise original_exception 2025-10-10T02:02:57.1413088Z utils.py:836: in wrapper 2025-10-10T02:02:57.1413281Z func(*args, **kwargs) 2025-10-10T02:02:57.1413530Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1413690Z LLM( 2025-10-10T02:02:57.1414018Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1414336Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1414717Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1415116Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1415488Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1416157Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1416525Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1417453Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1417919Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1418143Z super().__init__( 2025-10-10T02:02:57.1418478Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1419463Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1419747Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1420152Z next(self.gen) 2025-10-10T02:02:57.1420652Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1420966Z wait_for_engine_startup( 2025-10-10T02:02:57.1421078Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1421083Z 2025-10-10T02:02:57.1423282Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1423562Z "See root cause above. " 2025-10-10T02:02:57.1423932Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1424313Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1424400Z 2025-10-10T02:02:57.1424733Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1424999Z ____________ test_can_initialize_large_subset[TeleChat2ForCausalLM] ____________ 2025-10-10T02:02:57.1425004Z 2025-10-10T02:02:57.1425150Z model_arch = 'TeleChat2ForCausalLM' 2025-10-10T02:02:57.1425441Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa713260> 2025-10-10T02:02:57.1425445Z 2025-10-10T02:02:57.1429564Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1429896Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1430171Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1430417Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1430576Z  2025-10-10T02:02:57.1430909Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1431009Z  test. 2025-10-10T02:02:57.1431117Z  """ 2025-10-10T02:02:57.1431513Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1431886Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1432197Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1432202Z 2025-10-10T02:02:57.1432346Z models/test_initialization.py:131: 2025-10-10T02:02:57.1432450Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1432568Z utils.py:892: in wrapper 2025-10-10T02:02:57.1432735Z raise original_exception 2025-10-10T02:02:57.1432841Z utils.py:836: in wrapper 2025-10-10T02:02:57.1432967Z func(*args, **kwargs) 2025-10-10T02:02:57.1433161Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1433335Z LLM( 2025-10-10T02:02:57.1433661Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1433891Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1434292Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1435001Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1435356Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1436266Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1436632Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1437564Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1438008Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1438221Z super().__init__( 2025-10-10T02:02:57.1438545Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1439730Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1440160Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1440705Z next(self.gen) 2025-10-10T02:02:57.1441111Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1441429Z wait_for_engine_startup( 2025-10-10T02:02:57.1441632Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1441637Z 2025-10-10T02:02:57.1443707Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1444182Z "See root cause above. " 2025-10-10T02:02:57.1444634Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1445181Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1445241Z 2025-10-10T02:02:57.1445650Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1445913Z ______________ test_can_initialize_large_subset[MiDashengLMModel] ______________ 2025-10-10T02:02:57.1445918Z 2025-10-10T02:02:57.1446001Z model_arch = 'MiDashengLMModel' 2025-10-10T02:02:57.1446173Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71cfb0> 2025-10-10T02:02:57.1446177Z 2025-10-10T02:02:57.1450333Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1450651Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1450888Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1451223Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1451316Z  2025-10-10T02:02:57.1451578Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1451683Z  test. 2025-10-10T02:02:57.1451796Z  """ 2025-10-10T02:02:57.1452177Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1452602Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1452946Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1452956Z 2025-10-10T02:02:57.1453107Z models/test_initialization.py:131: 2025-10-10T02:02:57.1453218Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1453350Z utils.py:892: in wrapper 2025-10-10T02:02:57.1453552Z raise original_exception 2025-10-10T02:02:57.1453679Z utils.py:836: in wrapper 2025-10-10T02:02:57.1453819Z func(*args, **kwargs) 2025-10-10T02:02:57.1454086Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1454293Z LLM( 2025-10-10T02:02:57.1454722Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1454971Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1455476Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1455975Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1456434Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1457088Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1457645Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1458607Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1459095Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1459394Z super().__init__( 2025-10-10T02:02:57.1459745Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1460544Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1460982Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1461180Z next(self.gen) 2025-10-10T02:02:57.1461672Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1461818Z wait_for_engine_startup( 2025-10-10T02:02:57.1461923Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1461928Z 2025-10-10T02:02:57.1463857Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1464108Z "See root cause above. " 2025-10-10T02:02:57.1464597Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1464993Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1465004Z 2025-10-10T02:02:57.1465331Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1465571Z _________________ test_can_initialize_large_subset[Qwen2Model] _________________ 2025-10-10T02:02:57.1465575Z 2025-10-10T02:02:57.1465649Z model_arch = 'Qwen2Model' 2025-10-10T02:02:57.1465818Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f221242b8c0> 2025-10-10T02:02:57.1465822Z 2025-10-10T02:02:57.1470628Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1470973Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1471272Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1471508Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1471600Z  2025-10-10T02:02:57.1471863Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1471958Z  test. 2025-10-10T02:02:57.1472073Z  """ 2025-10-10T02:02:57.1472393Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1472871Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1473306Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1473315Z 2025-10-10T02:02:57.1473466Z models/test_initialization.py:131: 2025-10-10T02:02:57.1473568Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1473792Z utils.py:892: in wrapper 2025-10-10T02:02:57.1473965Z raise original_exception 2025-10-10T02:02:57.1474074Z utils.py:836: in wrapper 2025-10-10T02:02:57.1474196Z func(*args, **kwargs) 2025-10-10T02:02:57.1474506Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1474634Z LLM( 2025-10-10T02:02:57.1474956Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1475175Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1475745Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1476103Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1476568Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1477121Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1477480Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1478442Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1478896Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1479349Z super().__init__( 2025-10-10T02:02:57.1479707Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1480661Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1480956Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1481351Z next(self.gen) 2025-10-10T02:02:57.1481831Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1482143Z wait_for_engine_startup( 2025-10-10T02:02:57.1482246Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1482251Z 2025-10-10T02:02:57.1484391Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1484747Z "See root cause above. " 2025-10-10T02:02:57.1485156Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1485552Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1485556Z 2025-10-10T02:02:57.1485888Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1486144Z _________ test_can_initialize_large_subset[BertForTokenClassification] _________ 2025-10-10T02:02:57.1486147Z 2025-10-10T02:02:57.1486318Z model_arch = 'BertForTokenClassification' 2025-10-10T02:02:57.1486492Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71dac0> 2025-10-10T02:02:57.1486498Z 2025-10-10T02:02:57.1490672Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1491102Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1491368Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1491594Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1491682Z  2025-10-10T02:02:57.1492005Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1492172Z  test. 2025-10-10T02:02:57.1492286Z  """ 2025-10-10T02:02:57.1492741Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1493255Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1493571Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1493586Z 2025-10-10T02:02:57.1493748Z models/test_initialization.py:131: 2025-10-10T02:02:57.1493921Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1494068Z utils.py:892: in wrapper 2025-10-10T02:02:57.1494234Z raise original_exception 2025-10-10T02:02:57.1494397Z utils.py:836: in wrapper 2025-10-10T02:02:57.1494586Z func(*args, **kwargs) 2025-10-10T02:02:57.1494791Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1494983Z LLM( 2025-10-10T02:02:57.1495353Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1495695Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1496334Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1496599Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1497051Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1497472Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1497946Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1498935Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1499413Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1499631Z super().__init__( 2025-10-10T02:02:57.1500054Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1500926Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1501259Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1501641Z next(self.gen) 2025-10-10T02:02:57.1502141Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1502424Z wait_for_engine_startup( 2025-10-10T02:02:57.1502529Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1502534Z 2025-10-10T02:02:57.1504612Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1504999Z "See root cause above. " 2025-10-10T02:02:57.1505489Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1505878Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1505892Z 2025-10-10T02:02:57.1506222Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1506548Z ____________ test_can_initialize_large_subset[InternLM3ForCausalLM] ____________ 2025-10-10T02:02:57.1506553Z 2025-10-10T02:02:57.1506640Z model_arch = 'InternLM3ForCausalLM' 2025-10-10T02:02:57.1506877Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa75b980> 2025-10-10T02:02:57.1506881Z 2025-10-10T02:02:57.1510957Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1511284Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1511467Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1511693Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1511778Z  2025-10-10T02:02:57.1512034Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1512138Z  test. 2025-10-10T02:02:57.1512247Z  """ 2025-10-10T02:02:57.1512543Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1512881Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1513103Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1513116Z 2025-10-10T02:02:57.1513255Z models/test_initialization.py:131: 2025-10-10T02:02:57.1513351Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1513466Z utils.py:892: in wrapper 2025-10-10T02:02:57.1513630Z raise original_exception 2025-10-10T02:02:57.1513737Z utils.py:836: in wrapper 2025-10-10T02:02:57.1513863Z func(*args, **kwargs) 2025-10-10T02:02:57.1514042Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1514144Z LLM( 2025-10-10T02:02:57.1514461Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1515461Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1515823Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1516395Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1516737Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1517660Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1518074Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1519136Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1519468Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1519907Z super().__init__( 2025-10-10T02:02:57.1520250Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1521166Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1521396Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1522074Z next(self.gen) 2025-10-10T02:02:57.1522479Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1522858Z wait_for_engine_startup( 2025-10-10T02:02:57.1522975Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1522984Z 2025-10-10T02:02:57.1525136Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1525410Z "See root cause above. " 2025-10-10T02:02:57.1525772Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1526182Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1526193Z 2025-10-10T02:02:57.1526528Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1526788Z ___________ test_can_initialize_large_subset[EagleLlamaForCausalLM] ____________ 2025-10-10T02:02:57.1526792Z 2025-10-10T02:02:57.1526875Z model_arch = 'EagleLlamaForCausalLM' 2025-10-10T02:02:57.1527051Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa713200> 2025-10-10T02:02:57.1527055Z 2025-10-10T02:02:57.1531563Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1531912Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1532215Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1532543Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1532640Z  2025-10-10T02:02:57.1532904Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1533008Z  test. 2025-10-10T02:02:57.1533120Z  """ 2025-10-10T02:02:57.1533617Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1534160Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1534408Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1534413Z 2025-10-10T02:02:57.1534576Z models/test_initialization.py:131: 2025-10-10T02:02:57.1534756Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1534880Z utils.py:892: in wrapper 2025-10-10T02:02:57.1535049Z raise original_exception 2025-10-10T02:02:57.1535229Z utils.py:836: in wrapper 2025-10-10T02:02:57.1535457Z func(*args, **kwargs) 2025-10-10T02:02:57.1535694Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1535868Z LLM( 2025-10-10T02:02:57.1536196Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1536587Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1537018Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1537241Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1537768Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1538219Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1538584Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1539537Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1540084Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1540297Z super().__init__( 2025-10-10T02:02:57.1540649Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1541343Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1541621Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1542153Z next(self.gen) 2025-10-10T02:02:57.1542700Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1542859Z wait_for_engine_startup( 2025-10-10T02:02:57.1542961Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1542966Z 2025-10-10T02:02:57.1544855Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1545103Z "See root cause above. " 2025-10-10T02:02:57.1545677Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1546094Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1546099Z 2025-10-10T02:02:57.1546438Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1546708Z ___________ test_can_initialize_large_subset[LlamaForCausalLMEagle3] ___________ 2025-10-10T02:02:57.1546712Z 2025-10-10T02:02:57.1546801Z model_arch = 'LlamaForCausalLMEagle3' 2025-10-10T02:02:57.1546974Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa75a990> 2025-10-10T02:02:57.1546982Z 2025-10-10T02:02:57.1551413Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1551892Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1552245Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1552524Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1552625Z  2025-10-10T02:02:57.1552884Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1553055Z  test. 2025-10-10T02:02:57.1553200Z  """ 2025-10-10T02:02:57.1553674Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1554214Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1554453Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1554458Z 2025-10-10T02:02:57.1554679Z models/test_initialization.py:131: 2025-10-10T02:02:57.1554819Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1554943Z utils.py:892: in wrapper 2025-10-10T02:02:57.1555220Z raise original_exception 2025-10-10T02:02:57.1555433Z utils.py:836: in wrapper 2025-10-10T02:02:57.1555581Z func(*args, **kwargs) 2025-10-10T02:02:57.1555770Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1555868Z LLM( 2025-10-10T02:02:57.1556262Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1556629Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1557012Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1557239Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1557729Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1558031Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1558391Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1559518Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1560002Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1560497Z super().__init__( 2025-10-10T02:02:57.1560941Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1561580Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1561909Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1562352Z next(self.gen) 2025-10-10T02:02:57.1562871Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1563021Z wait_for_engine_startup( 2025-10-10T02:02:57.1563124Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1563129Z 2025-10-10T02:02:57.1565208Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1565662Z "See root cause above. " 2025-10-10T02:02:57.1566042Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1566436Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1566521Z 2025-10-10T02:02:57.1566853Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1567100Z ________________ test_can_initialize_large_subset[MiMoMTPModel] ________________ 2025-10-10T02:02:57.1567104Z 2025-10-10T02:02:57.1567185Z model_arch = 'MiMoMTPModel' 2025-10-10T02:02:57.1567365Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71f020> 2025-10-10T02:02:57.1567369Z 2025-10-10T02:02:57.1571548Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1571945Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1572291Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1572549Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1572642Z  2025-10-10T02:02:57.1572899Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1573001Z  test. 2025-10-10T02:02:57.1573109Z  """ 2025-10-10T02:02:57.1573555Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1574054Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1574301Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1574316Z 2025-10-10T02:02:57.1574460Z models/test_initialization.py:131: 2025-10-10T02:02:57.1574561Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1574687Z utils.py:892: in wrapper 2025-10-10T02:02:57.1574859Z raise original_exception 2025-10-10T02:02:57.1574965Z utils.py:836: in wrapper 2025-10-10T02:02:57.1575128Z func(*args, **kwargs) 2025-10-10T02:02:57.1575426Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1575536Z LLM( 2025-10-10T02:02:57.1575859Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1576076Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1576591Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1576852Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1577182Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1578101Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1578462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1579427Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1579877Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1580316Z super().__init__( 2025-10-10T02:02:57.1580664Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1581463Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1581852Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1582203Z next(self.gen) 2025-10-10T02:02:57.1582708Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1582864Z wait_for_engine_startup( 2025-10-10T02:02:57.1582963Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1582973Z 2025-10-10T02:02:57.1585113Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1585400Z "See root cause above. " 2025-10-10T02:02:57.1585760Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1586151Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1586156Z 2025-10-10T02:02:57.1586473Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1586717Z ______________ test_can_initialize_large_subset[GPTJForCausalLM] _______________ 2025-10-10T02:02:57.1586725Z 2025-10-10T02:02:57.1586806Z model_arch = 'GPTJForCausalLM' 2025-10-10T02:02:57.1586983Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa758620> 2025-10-10T02:02:57.1586986Z 2025-10-10T02:02:57.1591917Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1592243Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1592427Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1592694Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1592854Z  2025-10-10T02:02:57.1593134Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1593239Z  test. 2025-10-10T02:02:57.1593456Z  """ 2025-10-10T02:02:57.1593791Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1594132Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1594350Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1594361Z 2025-10-10T02:02:57.1594503Z models/test_initialization.py:131: 2025-10-10T02:02:57.1594674Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1594840Z utils.py:892: in wrapper 2025-10-10T02:02:57.1595006Z raise original_exception 2025-10-10T02:02:57.1595122Z utils.py:836: in wrapper 2025-10-10T02:02:57.1595251Z func(*args, **kwargs) 2025-10-10T02:02:57.1595521Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1595708Z LLM( 2025-10-10T02:02:57.1596279Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1596624Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1597005Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1597444Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1597801Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1598405Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1598869Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1599927Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1600346Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1600814Z super().__init__( 2025-10-10T02:02:57.1601177Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1601867Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1602170Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1602585Z next(self.gen) 2025-10-10T02:02:57.1602941Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1603083Z wait_for_engine_startup( 2025-10-10T02:02:57.1603184Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1603189Z 2025-10-10T02:02:57.1605965Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1606244Z "See root cause above. " 2025-10-10T02:02:57.1606608Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1606992Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1606996Z 2025-10-10T02:02:57.1607322Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1607604Z ____ test_can_initialize_large_subset[ModernBertForSequenceClassification] _____ 2025-10-10T02:02:57.1607608Z 2025-10-10T02:02:57.1607733Z model_arch = 'ModernBertForSequenceClassification' 2025-10-10T02:02:57.1607908Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71d010> 2025-10-10T02:02:57.1607911Z 2025-10-10T02:02:57.1612232Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1612559Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1612742Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1612972Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1613171Z  2025-10-10T02:02:57.1613433Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1613536Z  test. 2025-10-10T02:02:57.1613649Z  """ 2025-10-10T02:02:57.1613942Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1614448Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1614687Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1614691Z 2025-10-10T02:02:57.1614827Z models/test_initialization.py:131: 2025-10-10T02:02:57.1614923Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1615040Z utils.py:892: in wrapper 2025-10-10T02:02:57.1615214Z raise original_exception 2025-10-10T02:02:57.1615355Z utils.py:836: in wrapper 2025-10-10T02:02:57.1615640Z func(*args, **kwargs) 2025-10-10T02:02:57.1615885Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1615991Z LLM( 2025-10-10T02:02:57.1616316Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1616686Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1617061Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1617709Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1618094Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1618981Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1619396Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1620258Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1620606Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1620981Z super().__init__( 2025-10-10T02:02:57.1621465Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1622163Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1622493Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1622895Z next(self.gen) 2025-10-10T02:02:57.1623264Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1623538Z wait_for_engine_startup( 2025-10-10T02:02:57.1623661Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1623671Z 2025-10-10T02:02:57.1626100Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1626357Z "See root cause above. " 2025-10-10T02:02:57.1626713Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1627207Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1627221Z 2025-10-10T02:02:57.1627561Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1627807Z ______________ test_can_initialize_large_subset[GLM4VForCausalLM] ______________ 2025-10-10T02:02:57.1627854Z 2025-10-10T02:02:57.1627939Z model_arch = 'GLM4VForCausalLM' 2025-10-10T02:02:57.1628112Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa712ba0> 2025-10-10T02:02:57.1628116Z 2025-10-10T02:02:57.1632665Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1633176Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1633415Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1633718Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1633809Z  2025-10-10T02:02:57.1634115Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1634229Z  test. 2025-10-10T02:02:57.1634367Z  """ 2025-10-10T02:02:57.1634791Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1635279Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1635518Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1635522Z 2025-10-10T02:02:57.1635659Z models/test_initialization.py:131: 2025-10-10T02:02:57.1635767Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1635879Z utils.py:892: in wrapper 2025-10-10T02:02:57.1636048Z raise original_exception 2025-10-10T02:02:57.1636152Z utils.py:836: in wrapper 2025-10-10T02:02:57.1636304Z func(*args, **kwargs) 2025-10-10T02:02:57.1636588Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1636693Z LLM( 2025-10-10T02:02:57.1637019Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1637229Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1637735Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1637998Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1638328Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1639340Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1639704Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1640751Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1641234Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1641540Z super().__init__( 2025-10-10T02:02:57.1642024Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1642862Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1643126Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1643613Z next(self.gen) 2025-10-10T02:02:57.1644097Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1644406Z wait_for_engine_startup( 2025-10-10T02:02:57.1644520Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1645890Z 2025-10-10T02:02:57.1646522Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1646789Z "See root cause above. " 2025-10-10T02:02:57.1647226Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1647684Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1647695Z 2025-10-10T02:02:57.1648028Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1648313Z _____ test_can_initialize_large_subset[ChameleonForConditionalGeneration] ______ 2025-10-10T02:02:57.1648317Z 2025-10-10T02:02:57.1648438Z model_arch = 'ChameleonForConditionalGeneration' 2025-10-10T02:02:57.1648612Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71c500> 2025-10-10T02:02:57.1648616Z 2025-10-10T02:02:57.1652927Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1653330Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1653522Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1653746Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1653847Z  2025-10-10T02:02:57.1654207Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1654356Z  test. 2025-10-10T02:02:57.1654473Z  """ 2025-10-10T02:02:57.1654873Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1655247Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1655474Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1655481Z 2025-10-10T02:02:57.1655621Z models/test_initialization.py:131: 2025-10-10T02:02:57.1655735Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1655847Z utils.py:892: in wrapper 2025-10-10T02:02:57.1656142Z raise original_exception 2025-10-10T02:02:57.1656271Z utils.py:836: in wrapper 2025-10-10T02:02:57.1656390Z func(*args, **kwargs) 2025-10-10T02:02:57.1656572Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1656667Z LLM( 2025-10-10T02:02:57.1657031Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1657347Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1657811Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1658073Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1658554Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1659410Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1661191Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1661643Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1662073Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1662442Z super().__init__( 2025-10-10T02:02:57.1662982Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1663564Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1663966Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1664174Z next(self.gen) 2025-10-10T02:02:57.1664530Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1664703Z wait_for_engine_startup( 2025-10-10T02:02:57.1664886Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1664892Z 2025-10-10T02:02:57.1666144Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1666555Z "See root cause above. " 2025-10-10T02:02:57.1666937Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1667428Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1667441Z 2025-10-10T02:02:57.1667792Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1668053Z ___________ test_can_initialize_large_subset[GraniteMoeForCausalLM] ____________ 2025-10-10T02:02:57.1668057Z 2025-10-10T02:02:57.1668142Z model_arch = 'GraniteMoeForCausalLM' 2025-10-10T02:02:57.1668312Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa793140> 2025-10-10T02:02:57.1668323Z 2025-10-10T02:02:57.1672931Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1673431Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1673738Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1673989Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1674080Z  2025-10-10T02:02:57.1674338Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1674441Z  test. 2025-10-10T02:02:57.1674551Z  """ 2025-10-10T02:02:57.1675016Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1675644Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1676014Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1676020Z 2025-10-10T02:02:57.1676172Z models/test_initialization.py:131: 2025-10-10T02:02:57.1676341Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1676490Z utils.py:892: in wrapper 2025-10-10T02:02:57.1676790Z raise original_exception 2025-10-10T02:02:57.1676908Z utils.py:836: in wrapper 2025-10-10T02:02:57.1677032Z func(*args, **kwargs) 2025-10-10T02:02:57.1677205Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1677299Z LLM( 2025-10-10T02:02:57.1677746Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1678036Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1678601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1678958Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1679402Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1679708Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1680138Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1681043Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1681517Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1681707Z super().__init__( 2025-10-10T02:02:57.1682153Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1683101Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1683458Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1683717Z next(self.gen) 2025-10-10T02:02:57.1684182Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1684453Z wait_for_engine_startup( 2025-10-10T02:02:57.1684564Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1684575Z 2025-10-10T02:02:57.1686615Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1686917Z "See root cause above. " 2025-10-10T02:02:57.1687281Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1687671Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1687683Z 2025-10-10T02:02:57.1688007Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1688248Z _______________ test_can_initialize_large_subset[PhiForCausalLM] _______________ 2025-10-10T02:02:57.1688325Z 2025-10-10T02:02:57.1688406Z model_arch = 'PhiForCausalLM' 2025-10-10T02:02:57.1688578Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7581d0> 2025-10-10T02:02:57.1688583Z 2025-10-10T02:02:57.1693268Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1693856Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1694065Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1694293Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1694385Z  2025-10-10T02:02:57.1694841Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1694974Z  test. 2025-10-10T02:02:57.1695101Z  """ 2025-10-10T02:02:57.1696488Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1697105Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1697359Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1697376Z 2025-10-10T02:02:57.1697519Z models/test_initialization.py:131: 2025-10-10T02:02:57.1697617Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1697751Z utils.py:892: in wrapper 2025-10-10T02:02:57.1698054Z raise original_exception 2025-10-10T02:02:57.1698231Z utils.py:836: in wrapper 2025-10-10T02:02:57.1698367Z func(*args, **kwargs) 2025-10-10T02:02:57.1698548Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1698648Z LLM( 2025-10-10T02:02:57.1699018Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1699403Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1699784Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1699999Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1700383Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1700783Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1701152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1701684Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1702066Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1702248Z super().__init__( 2025-10-10T02:02:57.1703984Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1704415Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1704666Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1704843Z next(self.gen) 2025-10-10T02:02:57.1705209Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1705462Z wait_for_engine_startup( 2025-10-10T02:02:57.1705606Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1705616Z 2025-10-10T02:02:57.1707327Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1707693Z "See root cause above. " 2025-10-10T02:02:57.1708056Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1708441Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1708446Z 2025-10-10T02:02:57.1708769Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1709123Z _______ test_can_initialize_large_subset[Blip2ForConditionalGeneration] ________ 2025-10-10T02:02:57.1709175Z 2025-10-10T02:02:57.1709282Z model_arch = 'Blip2ForConditionalGeneration' 2025-10-10T02:02:57.1709466Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa792f00> 2025-10-10T02:02:57.1709473Z 2025-10-10T02:02:57.1713630Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1714019Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1714288Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1714518Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1714617Z  2025-10-10T02:02:57.1714884Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1714982Z  test. 2025-10-10T02:02:57.1715095Z  """ 2025-10-10T02:02:57.1715521Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1715935Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1716231Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1716236Z 2025-10-10T02:02:57.1716384Z models/test_initialization.py:131: 2025-10-10T02:02:57.1716485Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1716603Z utils.py:892: in wrapper 2025-10-10T02:02:57.1716782Z raise original_exception 2025-10-10T02:02:57.1716891Z utils.py:836: in wrapper 2025-10-10T02:02:57.1717035Z func(*args, **kwargs) 2025-10-10T02:02:57.1717320Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1717425Z LLM( 2025-10-10T02:02:57.1717759Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1717971Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1718451Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1718984Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1719473Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1720312Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1720685Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1721725Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1722283Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1722562Z super().__init__( 2025-10-10T02:02:57.1722978Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1723683Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1724018Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1724736Z next(self.gen) 2025-10-10T02:02:57.1725544Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1729835Z wait_for_engine_startup( 2025-10-10T02:02:57.1729986Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1729998Z 2025-10-10T02:02:57.1730479Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1730725Z "See root cause above. " 2025-10-10T02:02:57.1731090Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1731489Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1731495Z 2025-10-10T02:02:57.1731834Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1732081Z _____________ test_can_initialize_large_subset[MixtralForCausalLM] _____________ 2025-10-10T02:02:57.1732087Z 2025-10-10T02:02:57.1732170Z model_arch = 'MixtralForCausalLM' 2025-10-10T02:02:57.1732346Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71fb90> 2025-10-10T02:02:57.1732350Z 2025-10-10T02:02:57.1734023Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1734473Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1734681Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1734913Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1735009Z  2025-10-10T02:02:57.1735266Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1735365Z  test. 2025-10-10T02:02:57.1735496Z  """ 2025-10-10T02:02:57.1735904Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1736358Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1736594Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1736606Z 2025-10-10T02:02:57.1736849Z models/test_initialization.py:131: 2025-10-10T02:02:57.1736951Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1737063Z utils.py:892: in wrapper 2025-10-10T02:02:57.1737247Z raise original_exception 2025-10-10T02:02:57.1737370Z utils.py:836: in wrapper 2025-10-10T02:02:57.1737591Z func(*args, **kwargs) 2025-10-10T02:02:57.1737852Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1737958Z LLM( 2025-10-10T02:02:57.1738289Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1738589Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1738998Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1739340Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1739862Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1740565Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1740917Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1741917Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1742377Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1742820Z super().__init__( 2025-10-10T02:02:57.1743280Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1744039Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1744350Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1744853Z next(self.gen) 2025-10-10T02:02:57.1745212Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1745351Z wait_for_engine_startup( 2025-10-10T02:02:57.1745454Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1745459Z 2025-10-10T02:02:57.1747534Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1747880Z "See root cause above. " 2025-10-10T02:02:57.1748243Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1748633Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1748638Z 2025-10-10T02:02:57.1748959Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1749227Z ___ test_can_initialize_large_subset[Ernie4_5_VLMoeForConditionalGeneration] ___ 2025-10-10T02:02:57.1749230Z 2025-10-10T02:02:57.1749361Z model_arch = 'Ernie4_5_VLMoeForConditionalGeneration' 2025-10-10T02:02:57.1749536Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa792f00> 2025-10-10T02:02:57.1749539Z 2025-10-10T02:02:57.1754019Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1754789Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1755048Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1755342Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1755655Z  2025-10-10T02:02:57.1756193Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1756436Z  test. 2025-10-10T02:02:57.1756635Z  """ 2025-10-10T02:02:57.1757018Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1757585Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1758082Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1758090Z 2025-10-10T02:02:57.1758399Z models/test_initialization.py:131: 2025-10-10T02:02:57.1758625Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1758900Z utils.py:892: in wrapper 2025-10-10T02:02:57.1759273Z raise original_exception 2025-10-10T02:02:57.1759413Z utils.py:836: in wrapper 2025-10-10T02:02:57.1759568Z func(*args, **kwargs) 2025-10-10T02:02:57.1759860Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1760089Z LLM( 2025-10-10T02:02:57.1760616Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1760891Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1761599Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1761910Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1762571Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1762878Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1763457Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1763899Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1764484Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1764694Z super().__init__( 2025-10-10T02:02:57.1765237Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1765502Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1765734Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1765907Z next(self.gen) 2025-10-10T02:02:57.1766284Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1766518Z wait_for_engine_startup( 2025-10-10T02:02:57.1766696Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1766703Z 2025-10-10T02:02:57.1767770Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1768209Z "See root cause above. " 2025-10-10T02:02:57.1768697Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1769173Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1769178Z 2025-10-10T02:02:57.1769502Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1769743Z _________________ test_can_initialize_large_subset[Terratorch] _________________ 2025-10-10T02:02:57.1769747Z 2025-10-10T02:02:57.1769823Z model_arch = 'Terratorch' 2025-10-10T02:02:57.1769996Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa758c50> 2025-10-10T02:02:57.1770000Z 2025-10-10T02:02:57.1774977Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1775441Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1775648Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1775951Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1776101Z  2025-10-10T02:02:57.1776505Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1776615Z  test. 2025-10-10T02:02:57.1776755Z  """ 2025-10-10T02:02:57.1777177Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1777548Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1777774Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1777780Z 2025-10-10T02:02:57.1777924Z models/test_initialization.py:131: 2025-10-10T02:02:57.1778087Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1778269Z utils.py:892: in wrapper 2025-10-10T02:02:57.1778454Z raise original_exception 2025-10-10T02:02:57.1778566Z utils.py:836: in wrapper 2025-10-10T02:02:57.1778749Z func(*args, **kwargs) 2025-10-10T02:02:57.1779082Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1779252Z LLM( 2025-10-10T02:02:57.1779699Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1780036Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1780659Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1780900Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1781349Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1781743Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1782320Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1782788Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1783267Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1783620Z super().__init__( 2025-10-10T02:02:57.1784120Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1784723Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1785103Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1785410Z next(self.gen) 2025-10-10T02:02:57.1785787Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1785923Z wait_for_engine_startup( 2025-10-10T02:02:57.1786031Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1786036Z 2025-10-10T02:02:57.1787813Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1788075Z "See root cause above. " 2025-10-10T02:02:57.1788606Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1788941Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} 2025-10-10T02:02:57.1788946Z 2025-10-10T02:02:57.1789267Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1789529Z __________ test_can_initialize_large_subset[MiniMaxText01ForCausalLM] __________ 2025-10-10T02:02:57.1789537Z 2025-10-10T02:02:57.1789628Z model_arch = 'MiniMaxText01ForCausalLM' 2025-10-10T02:02:57.1789802Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa791cd0> 2025-10-10T02:02:57.1789807Z 2025-10-10T02:02:57.1794253Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1794580Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1794806Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1795155Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1795250Z  2025-10-10T02:02:57.1795515Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1795616Z  test. 2025-10-10T02:02:57.1795785Z  """ 2025-10-10T02:02:57.1796462Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1796935Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1797212Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1797227Z 2025-10-10T02:02:57.1797370Z models/test_initialization.py:131: 2025-10-10T02:02:57.1797468Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1797577Z utils.py:892: in wrapper 2025-10-10T02:02:57.1797767Z raise original_exception 2025-10-10T02:02:57.1797965Z utils.py:836: in wrapper 2025-10-10T02:02:57.1798349Z func(*args, **kwargs) 2025-10-10T02:02:57.1798654Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1798759Z LLM( 2025-10-10T02:02:57.1799237Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1799566Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1800051Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1800277Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1800738Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1801250Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1801595Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1802661Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1803123Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1803312Z super().__init__( 2025-10-10T02:02:57.1803705Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1804534Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1804896Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1805598Z next(self.gen) 2025-10-10T02:02:57.1805975Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1806109Z wait_for_engine_startup( 2025-10-10T02:02:57.1806214Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1806219Z 2025-10-10T02:02:57.1808030Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1808400Z "See root cause above. " 2025-10-10T02:02:57.1808778Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1809166Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1809175Z 2025-10-10T02:02:57.1809497Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1809747Z _____________ test_can_initialize_large_subset[Cohere2ForCausalLM] _____________ 2025-10-10T02:02:57.1809751Z 2025-10-10T02:02:57.1809834Z model_arch = 'Cohere2ForCausalLM' 2025-10-10T02:02:57.1810009Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71fa10> 2025-10-10T02:02:57.1810015Z 2025-10-10T02:02:57.1814636Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1814960Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1815161Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1815506Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1815680Z  2025-10-10T02:02:57.1815937Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1816040Z  test. 2025-10-10T02:02:57.1816150Z  """ 2025-10-10T02:02:57.1816527Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1817005Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1817314Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1817319Z 2025-10-10T02:02:57.1817467Z models/test_initialization.py:131: 2025-10-10T02:02:57.1817563Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1817678Z utils.py:892: in wrapper 2025-10-10T02:02:57.1817908Z raise original_exception 2025-10-10T02:02:57.1818017Z utils.py:836: in wrapper 2025-10-10T02:02:57.1818179Z func(*args, **kwargs) 2025-10-10T02:02:57.1818475Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1818602Z LLM( 2025-10-10T02:02:57.1818930Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1819167Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1819665Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1820217Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1820650Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1821369Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1821711Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1822746Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1823227Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1823535Z super().__init__( 2025-10-10T02:02:57.1823911Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1824692Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1825048Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1825750Z next(self.gen) 2025-10-10T02:02:57.1826131Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1826265Z wait_for_engine_startup( 2025-10-10T02:02:57.1826365Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1826371Z 2025-10-10T02:02:57.1828301Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1828670Z "See root cause above. " 2025-10-10T02:02:57.1829038Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1829497Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1829502Z 2025-10-10T02:02:57.1829823Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1830080Z ___________ test_can_initialize_large_subset[EagleLlama4ForCausalLM] ___________ 2025-10-10T02:02:57.1830151Z 2025-10-10T02:02:57.1830240Z model_arch = 'EagleLlama4ForCausalLM' 2025-10-10T02:02:57.1830413Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b6e10> 2025-10-10T02:02:57.1830417Z 2025-10-10T02:02:57.1834786Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1835106Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1835422Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1835754Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1835852Z  2025-10-10T02:02:57.1836107Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1836207Z  test. 2025-10-10T02:02:57.1836322Z  """ 2025-10-10T02:02:57.1836751Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1837200Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1837433Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1837442Z 2025-10-10T02:02:57.1837591Z models/test_initialization.py:131: 2025-10-10T02:02:57.1837693Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1837805Z utils.py:892: in wrapper 2025-10-10T02:02:57.1837971Z raise original_exception 2025-10-10T02:02:57.1838077Z utils.py:836: in wrapper 2025-10-10T02:02:57.1838209Z func(*args, **kwargs) 2025-10-10T02:02:57.1838513Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1838619Z LLM( 2025-10-10T02:02:57.1838939Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1839467Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1839860Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1840457Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1840812Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1841550Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1841907Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1843034Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1843392Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1843922Z super().__init__( 2025-10-10T02:02:57.1844284Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1845079Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1845434Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1845908Z next(self.gen) 2025-10-10T02:02:57.1846399Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1846657Z wait_for_engine_startup( 2025-10-10T02:02:57.1846757Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1846767Z 2025-10-10T02:02:57.1848652Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1848902Z "See root cause above. " 2025-10-10T02:02:57.1849454Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1849904Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1849913Z 2025-10-10T02:02:57.1850250Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1850503Z ____________ test_can_initialize_large_subset[NemotronForCausalLM] _____________ 2025-10-10T02:02:57.1850508Z 2025-10-10T02:02:57.1850595Z model_arch = 'NemotronForCausalLM' 2025-10-10T02:02:57.1850767Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa75b470> 2025-10-10T02:02:57.1850771Z 2025-10-10T02:02:57.1855164Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1855499Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1855825Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1856073Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1856164Z  2025-10-10T02:02:57.1856425Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1856524Z  test. 2025-10-10T02:02:57.1856637Z  """ 2025-10-10T02:02:57.1857006Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1857511Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1857786Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1857792Z 2025-10-10T02:02:57.1857940Z models/test_initialization.py:131: 2025-10-10T02:02:57.1858044Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1858162Z utils.py:892: in wrapper 2025-10-10T02:02:57.1858328Z raise original_exception 2025-10-10T02:02:57.1858434Z utils.py:836: in wrapper 2025-10-10T02:02:57.1858551Z func(*args, **kwargs) 2025-10-10T02:02:57.1858774Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1858951Z LLM( 2025-10-10T02:02:57.1859296Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1859586Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1860082Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1860428Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1860766Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1861677Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1862039Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1862982Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1863359Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1863858Z super().__init__( 2025-10-10T02:02:57.1864533Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1865317Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1865694Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1866006Z next(self.gen) 2025-10-10T02:02:57.1866384Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1866518Z wait_for_engine_startup( 2025-10-10T02:02:57.1866615Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1866620Z 2025-10-10T02:02:57.1868636Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1869026Z "See root cause above. " 2025-10-10T02:02:57.1869394Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1869785Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1869790Z 2025-10-10T02:02:57.1870118Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1870358Z ______________ test_can_initialize_large_subset[DbrxForCausalLM] _______________ 2025-10-10T02:02:57.1870361Z 2025-10-10T02:02:57.1870442Z model_arch = 'DbrxForCausalLM' 2025-10-10T02:02:57.1870613Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b68d0> 2025-10-10T02:02:57.1870616Z 2025-10-10T02:02:57.1875253Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1875689Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1875899Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1876133Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1876219Z  2025-10-10T02:02:57.1876482Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1876578Z  test. 2025-10-10T02:02:57.1876686Z  """ 2025-10-10T02:02:57.1877218Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1877677Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1877918Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1877989Z 2025-10-10T02:02:57.1878133Z models/test_initialization.py:131: 2025-10-10T02:02:57.1878234Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1878346Z utils.py:892: in wrapper 2025-10-10T02:02:57.1878509Z raise original_exception 2025-10-10T02:02:57.1878616Z utils.py:836: in wrapper 2025-10-10T02:02:57.1878777Z func(*args, **kwargs) 2025-10-10T02:02:57.1879155Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1879267Z LLM( 2025-10-10T02:02:57.1879648Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1879913Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1880442Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1880870Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1881287Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1882016Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1882369Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1883349Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1883853Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1884170Z super().__init__( 2025-10-10T02:02:57.1884551Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1885381Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1885724Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1886195Z next(self.gen) 2025-10-10T02:02:57.1886689Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1887014Z wait_for_engine_startup( 2025-10-10T02:02:57.1887129Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1887133Z 2025-10-10T02:02:57.1889027Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1889290Z "See root cause above. " 2025-10-10T02:02:57.1889650Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1890036Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1890041Z 2025-10-10T02:02:57.1890365Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1890732Z _______ test_can_initialize_large_subset[JambaForSequenceClassification] _______ 2025-10-10T02:02:57.1890736Z 2025-10-10T02:02:57.1890847Z model_arch = 'JambaForSequenceClassification' 2025-10-10T02:02:57.1891023Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa792c00> 2025-10-10T02:02:57.1891026Z 2025-10-10T02:02:57.1895346Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1895738Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1895921Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1896425Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1896536Z  2025-10-10T02:02:57.1896799Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1896901Z  test. 2025-10-10T02:02:57.1897106Z  """ 2025-10-10T02:02:57.1897459Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1897804Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1898025Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1898030Z 2025-10-10T02:02:57.1898169Z models/test_initialization.py:131: 2025-10-10T02:02:57.1898265Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1898378Z utils.py:892: in wrapper 2025-10-10T02:02:57.1898542Z raise original_exception 2025-10-10T02:02:57.1898646Z utils.py:836: in wrapper 2025-10-10T02:02:57.1898764Z func(*args, **kwargs) 2025-10-10T02:02:57.1898942Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1899050Z LLM( 2025-10-10T02:02:57.1899378Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1900145Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1900508Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1901190Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1901534Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1902440Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1902797Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1903749Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1904087Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1904608Z super().__init__( 2025-10-10T02:02:57.1905109Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1905982Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1906234Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1906957Z next(self.gen) 2025-10-10T02:02:57.1907462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1907596Z wait_for_engine_startup( 2025-10-10T02:02:57.1907695Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1907700Z 2025-10-10T02:02:57.1909706Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1910078Z "See root cause above. " 2025-10-10T02:02:57.1910434Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1910821Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1910829Z 2025-10-10T02:02:57.1911198Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1911512Z _______ test_can_initialize_large_subset[KimiVLForConditionalGeneration] _______ 2025-10-10T02:02:57.1911516Z 2025-10-10T02:02:57.1911623Z model_arch = 'KimiVLForConditionalGeneration' 2025-10-10T02:02:57.1911801Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b5160> 2025-10-10T02:02:57.1911804Z 2025-10-10T02:02:57.1916085Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1916567Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1916768Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1916996Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1917084Z  2025-10-10T02:02:57.1917347Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1917526Z  test. 2025-10-10T02:02:57.1917698Z  """ 2025-10-10T02:02:57.1918004Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1918487Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1918717Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1918731Z 2025-10-10T02:02:57.1918871Z models/test_initialization.py:131: 2025-10-10T02:02:57.1918969Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1919196Z utils.py:892: in wrapper 2025-10-10T02:02:57.1919364Z raise original_exception 2025-10-10T02:02:57.1919515Z utils.py:836: in wrapper 2025-10-10T02:02:57.1919741Z func(*args, **kwargs) 2025-10-10T02:02:57.1919934Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1920045Z LLM( 2025-10-10T02:02:57.1920364Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1920671Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1921125Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1921744Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1922311Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1922927Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1923405Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1924461Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1924894Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1925086Z super().__init__( 2025-10-10T02:02:57.1925569Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1926548Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1926804Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1927308Z next(self.gen) 2025-10-10T02:02:57.1927720Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1927859Z wait_for_engine_startup( 2025-10-10T02:02:57.1927960Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1927964Z 2025-10-10T02:02:57.1930216Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1930477Z "See root cause above. " 2025-10-10T02:02:57.1930839Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1931233Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1931238Z 2025-10-10T02:02:57.1931574Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1931809Z __________________ test_can_initialize_large_subset[Ovis2_5] ___________________ 2025-10-10T02:02:57.1931813Z 2025-10-10T02:02:57.1931882Z model_arch = 'Ovis2_5' 2025-10-10T02:02:57.1932060Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b9be0> 2025-10-10T02:02:57.1932064Z 2025-10-10T02:02:57.1936492Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1936966Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1937235Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1937542Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1937630Z  2025-10-10T02:02:57.1937890Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1938018Z  test. 2025-10-10T02:02:57.1938213Z  """ 2025-10-10T02:02:57.1938641Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1938991Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1939213Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1939310Z 2025-10-10T02:02:57.1939555Z models/test_initialization.py:131: 2025-10-10T02:02:57.1939664Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1939783Z utils.py:892: in wrapper 2025-10-10T02:02:57.1939945Z raise original_exception 2025-10-10T02:02:57.1940112Z utils.py:836: in wrapper 2025-10-10T02:02:57.1940228Z func(*args, **kwargs) 2025-10-10T02:02:57.1940531Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1940636Z LLM( 2025-10-10T02:02:57.1940953Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1941198Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1941672Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1941962Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1942482Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1943166Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1943566Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1944458Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1944847Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1945190Z super().__init__( 2025-10-10T02:02:57.1945519Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1947265Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1947643Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1947930Z next(self.gen) 2025-10-10T02:02:57.1948304Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1948435Z wait_for_engine_startup( 2025-10-10T02:02:57.1948562Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1948568Z 2025-10-10T02:02:57.1950146Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1950531Z "See root cause above. " 2025-10-10T02:02:57.1951016Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1951403Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1951410Z 2025-10-10T02:02:57.1951867Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1952121Z _____________ test_can_initialize_large_subset[Phi4MMForCausalLM] ______________ 2025-10-10T02:02:57.1952125Z 2025-10-10T02:02:57.1952210Z model_arch = 'Phi4MMForCausalLM' 2025-10-10T02:02:57.1952381Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa71dd00> 2025-10-10T02:02:57.1952385Z 2025-10-10T02:02:57.1956637Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1957206Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1957431Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1957759Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1958084Z  2025-10-10T02:02:57.1958455Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1958602Z  test. 2025-10-10T02:02:57.1958814Z  """ 2025-10-10T02:02:57.1959211Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1959646Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1960042Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1960101Z 2025-10-10T02:02:57.1960303Z models/test_initialization.py:131: 2025-10-10T02:02:57.1960401Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1960523Z utils.py:892: in wrapper 2025-10-10T02:02:57.1960685Z raise original_exception 2025-10-10T02:02:57.1960840Z utils.py:836: in wrapper 2025-10-10T02:02:57.1961041Z func(*args, **kwargs) 2025-10-10T02:02:57.1961293Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1961396Z LLM( 2025-10-10T02:02:57.1961800Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1962181Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1962663Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1962972Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1963472Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1963800Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1964278Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1964975Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1965421Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1965651Z super().__init__( 2025-10-10T02:02:57.1965978Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1966925Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1967171Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1967786Z next(self.gen) 2025-10-10T02:02:57.1968197Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1968456Z wait_for_engine_startup( 2025-10-10T02:02:57.1968635Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1968643Z 2025-10-10T02:02:57.1970511Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1970770Z "See root cause above. " 2025-10-10T02:02:57.1971127Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1971602Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1971608Z 2025-10-10T02:02:57.1971947Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1972192Z ____________ test_can_initialize_large_subset[Qwen3MoeForCausalLM] _____________ 2025-10-10T02:02:57.1972196Z 2025-10-10T02:02:57.1972279Z model_arch = 'Qwen3MoeForCausalLM' 2025-10-10T02:02:57.1972455Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7ba120> 2025-10-10T02:02:57.1972505Z 2025-10-10T02:02:57.1977222Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1977665Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1977909Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1978244Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1978407Z  2025-10-10T02:02:57.1978825Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1978990Z  test. 2025-10-10T02:02:57.1979197Z  """ 2025-10-10T02:02:57.1979583Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1980064Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1980496Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1980511Z 2025-10-10T02:02:57.1980670Z models/test_initialization.py:131: 2025-10-10T02:02:57.1980771Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1980888Z utils.py:892: in wrapper 2025-10-10T02:02:57.1981141Z raise original_exception 2025-10-10T02:02:57.1981312Z utils.py:836: in wrapper 2025-10-10T02:02:57.1981526Z func(*args, **kwargs) 2025-10-10T02:02:57.1981723Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.1981826Z LLM( 2025-10-10T02:02:57.1982324Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.1982613Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.1983100Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.1983334Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.1983783Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.1984050Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.1984484Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.1985067Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.1985530Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.1985739Z super().__init__( 2025-10-10T02:02:57.1986292Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.1986795Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.1987144Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.1987516Z next(self.gen) 2025-10-10T02:02:57.1987970Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.1988233Z wait_for_engine_startup( 2025-10-10T02:02:57.1988409Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.1988420Z 2025-10-10T02:02:57.1990346Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.1990729Z "See root cause above. " 2025-10-10T02:02:57.1991222Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.1991618Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.1991623Z 2025-10-10T02:02:57.1992084Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.1992344Z _____________ test_can_initialize_large_subset[RobertaForMaskedLM] _____________ 2025-10-10T02:02:57.1992351Z 2025-10-10T02:02:57.1992436Z model_arch = 'RobertaForMaskedLM' 2025-10-10T02:02:57.1992610Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7fb680> 2025-10-10T02:02:57.1992613Z 2025-10-10T02:02:57.1997003Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.1997429Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.1997624Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.1997967Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.1998059Z  2025-10-10T02:02:57.1998318Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.1998417Z  test. 2025-10-10T02:02:57.1998574Z  """ 2025-10-10T02:02:57.1999130Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.1999509Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.1999781Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.1999787Z 2025-10-10T02:02:57.2000015Z models/test_initialization.py:131: 2025-10-10T02:02:57.2000120Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2000235Z utils.py:892: in wrapper 2025-10-10T02:02:57.2000400Z raise original_exception 2025-10-10T02:02:57.2000611Z utils.py:836: in wrapper 2025-10-10T02:02:57.2000731Z func(*args, **kwargs) 2025-10-10T02:02:57.2000911Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2001070Z LLM( 2025-10-10T02:02:57.2001479Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2001784Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2002295Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2002565Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2003023Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2003750Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2004266Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2005328Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2005828Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2006034Z super().__init__( 2025-10-10T02:02:57.2006360Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2007178Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2007404Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2008034Z next(self.gen) 2025-10-10T02:02:57.2008391Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2008781Z wait_for_engine_startup( 2025-10-10T02:02:57.2008887Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2008892Z 2025-10-10T02:02:57.2010728Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2011109Z "See root cause above. " 2025-10-10T02:02:57.2011587Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2011976Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2011990Z 2025-10-10T02:02:57.2012440Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2012708Z __________ test_can_initialize_large_subset[LongcatFlashForCausalLM] ___________ 2025-10-10T02:02:57.2012713Z 2025-10-10T02:02:57.2012804Z model_arch = 'LongcatFlashForCausalLM' 2025-10-10T02:02:57.2012977Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b4fe0> 2025-10-10T02:02:57.2012981Z 2025-10-10T02:02:57.2017096Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2017547Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2017742Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2018120Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2018257Z  2025-10-10T02:02:57.2018526Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2018623Z  test. 2025-10-10T02:02:57.2018733Z  """ 2025-10-10T02:02:57.2019254Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2019684Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2019911Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2019915Z 2025-10-10T02:02:57.2020050Z models/test_initialization.py:131: 2025-10-10T02:02:57.2020155Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2020266Z utils.py:892: in wrapper 2025-10-10T02:02:57.2020611Z raise original_exception 2025-10-10T02:02:57.2020782Z utils.py:836: in wrapper 2025-10-10T02:02:57.2020903Z func(*args, **kwargs) 2025-10-10T02:02:57.2021081Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2021179Z LLM( 2025-10-10T02:02:57.2021604Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2021886Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2022350Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2022877Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2023278Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2023784Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2024157Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2025611Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2025969Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2026193Z super().__init__( 2025-10-10T02:02:57.2026771Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2027594Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2028052Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2028298Z next(self.gen) 2025-10-10T02:02:57.2028868Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2029022Z wait_for_engine_startup( 2025-10-10T02:02:57.2029127Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2029133Z 2025-10-10T02:02:57.2030963Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2031333Z "See root cause above. " 2025-10-10T02:02:57.2031797Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2032285Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2032294Z 2025-10-10T02:02:57.2032621Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2032915Z _____________ test_can_initialize_large_subset[Gemma2ForCausalLM] ______________ 2025-10-10T02:02:57.2032918Z 2025-10-10T02:02:57.2033001Z model_arch = 'Gemma2ForCausalLM' 2025-10-10T02:02:57.2033179Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7fa750> 2025-10-10T02:02:57.2033183Z 2025-10-10T02:02:57.2037342Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2037672Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2037976Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2038438Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2038544Z  2025-10-10T02:02:57.2038810Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2038909Z  test. 2025-10-10T02:02:57.2039182Z  """ 2025-10-10T02:02:57.2039739Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2040211Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2040443Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2040453Z 2025-10-10T02:02:57.2040592Z models/test_initialization.py:131: 2025-10-10T02:02:57.2040694Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2040804Z utils.py:892: in wrapper 2025-10-10T02:02:57.2040975Z raise original_exception 2025-10-10T02:02:57.2041159Z utils.py:836: in wrapper 2025-10-10T02:02:57.2041392Z func(*args, **kwargs) 2025-10-10T02:02:57.2041654Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2041834Z LLM( 2025-10-10T02:02:57.2042289Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2042660Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2043140Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2043444Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2044034Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2044300Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2044681Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2045414Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2045930Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2046245Z super().__init__( 2025-10-10T02:02:57.2046691Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2047641Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2048097Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2048413Z next(self.gen) 2025-10-10T02:02:57.2049075Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2049224Z wait_for_engine_startup( 2025-10-10T02:02:57.2049324Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2049330Z 2025-10-10T02:02:57.2051237Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2051694Z "See root cause above. " 2025-10-10T02:02:57.2052212Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2052607Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2052615Z 2025-10-10T02:02:57.2052940Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2053215Z _________ test_can_initialize_large_subset[Phi4MultimodalForCausalLM] __________ 2025-10-10T02:02:57.2053223Z 2025-10-10T02:02:57.2053315Z model_arch = 'Phi4MultimodalForCausalLM' 2025-10-10T02:02:57.2053486Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b6ea0> 2025-10-10T02:02:57.2053491Z 2025-10-10T02:02:57.2057590Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2057935Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2058276Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2058574Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2058663Z  2025-10-10T02:02:57.2058923Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2059020Z  test. 2025-10-10T02:02:57.2059131Z  """ 2025-10-10T02:02:57.2059528Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2060159Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2060408Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2060413Z 2025-10-10T02:02:57.2060572Z models/test_initialization.py:131: 2025-10-10T02:02:57.2060667Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2060778Z utils.py:892: in wrapper 2025-10-10T02:02:57.2060947Z raise original_exception 2025-10-10T02:02:57.2061054Z utils.py:836: in wrapper 2025-10-10T02:02:57.2061217Z func(*args, **kwargs) 2025-10-10T02:02:57.2061548Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2061681Z LLM( 2025-10-10T02:02:57.2062132Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2062651Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2063165Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2063438Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2064098Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2064389Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2064777Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2065488Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2066111Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2066428Z super().__init__( 2025-10-10T02:02:57.2066956Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2067598Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2068051Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2068367Z next(self.gen) 2025-10-10T02:02:57.2068835Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2068974Z wait_for_engine_startup( 2025-10-10T02:02:57.2069075Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2069086Z 2025-10-10T02:02:57.2070872Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2071139Z "See root cause above. " 2025-10-10T02:02:57.2071720Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2072223Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2072228Z 2025-10-10T02:02:57.2072559Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2072834Z ______ test_can_initialize_large_subset[Glm4vMoeForConditionalGeneration] ______ 2025-10-10T02:02:57.2072846Z 2025-10-10T02:02:57.2072957Z model_arch = 'Glm4vMoeForConditionalGeneration' 2025-10-10T02:02:57.2073130Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7f8920> 2025-10-10T02:02:57.2073134Z 2025-10-10T02:02:57.2077341Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2077681Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2077866Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2078089Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2078179Z  2025-10-10T02:02:57.2078502Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2078657Z  test. 2025-10-10T02:02:57.2078858Z  """ 2025-10-10T02:02:57.2079288Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2079639Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2080010Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2080075Z 2025-10-10T02:02:57.2080352Z models/test_initialization.py:131: 2025-10-10T02:02:57.2080462Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2080578Z utils.py:892: in wrapper 2025-10-10T02:02:57.2080753Z raise original_exception 2025-10-10T02:02:57.2080856Z utils.py:836: in wrapper 2025-10-10T02:02:57.2080978Z func(*args, **kwargs) 2025-10-10T02:02:57.2081156Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2081307Z LLM( 2025-10-10T02:02:57.2081788Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2082032Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2082399Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2082880Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2083239Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2084096Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2084451Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2085473Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2085821Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2086162Z super().__init__( 2025-10-10T02:02:57.2086503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2087502Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2087742Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2088272Z next(self.gen) 2025-10-10T02:02:57.2088763Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2088976Z wait_for_engine_startup( 2025-10-10T02:02:57.2089081Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2089086Z 2025-10-10T02:02:57.2091195Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2091459Z "See root cause above. " 2025-10-10T02:02:57.2091814Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2092319Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2092325Z 2025-10-10T02:02:57.2092651Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2092983Z _________________ test_can_initialize_large_subset[BertModel] __________________ 2025-10-10T02:02:57.2092987Z 2025-10-10T02:02:57.2093058Z model_arch = 'BertModel' 2025-10-10T02:02:57.2093226Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b5910> 2025-10-10T02:02:57.2093233Z 2025-10-10T02:02:57.2098347Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2098670Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2098859Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2099087Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2099183Z  2025-10-10T02:02:57.2099435Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2099654Z  test. 2025-10-10T02:02:57.2099847Z  """ 2025-10-10T02:02:57.2100229Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2100870Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2101313Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2101321Z 2025-10-10T02:02:57.2101476Z models/test_initialization.py:131: 2025-10-10T02:02:57.2101575Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2101686Z utils.py:892: in wrapper 2025-10-10T02:02:57.2101862Z raise original_exception 2025-10-10T02:02:57.2102002Z utils.py:836: in wrapper 2025-10-10T02:02:57.2102229Z func(*args, **kwargs) 2025-10-10T02:02:57.2102506Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2102614Z LLM( 2025-10-10T02:02:57.2102936Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2103246Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2103896Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2104186Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2104736Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2105362Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2105748Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2106678Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2107139Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2107690Z super().__init__( 2025-10-10T02:02:57.2108107Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2108781Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2109085Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2109712Z next(self.gen) 2025-10-10T02:02:57.2110379Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2110622Z wait_for_engine_startup( 2025-10-10T02:02:57.2110794Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2110937Z 2025-10-10T02:02:57.2112519Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2112932Z "See root cause above. " 2025-10-10T02:02:57.2113560Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2114191Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2114203Z 2025-10-10T02:02:57.2114640Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2114894Z ___________________ test_can_initialize_large_subset[GritLM] ___________________ 2025-10-10T02:02:57.2114904Z 2025-10-10T02:02:57.2114971Z model_arch = 'GritLM' 2025-10-10T02:02:57.2115149Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7b5eb0> 2025-10-10T02:02:57.2115153Z 2025-10-10T02:02:57.2118932Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2119366Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2119554Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2119792Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2119879Z  2025-10-10T02:02:57.2120225Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2120418Z  test. 2025-10-10T02:02:57.2120592Z  """ 2025-10-10T02:02:57.2121039Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2121479Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2121874Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2121889Z 2025-10-10T02:02:57.2122044Z models/test_initialization.py:131: 2025-10-10T02:02:57.2122149Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2122270Z utils.py:892: in wrapper 2025-10-10T02:02:57.2122437Z raise original_exception 2025-10-10T02:02:57.2122542Z utils.py:836: in wrapper 2025-10-10T02:02:57.2122665Z func(*args, **kwargs) 2025-10-10T02:02:57.2122987Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2123160Z LLM( 2025-10-10T02:02:57.2123484Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2123699Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2124275Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2124521Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2124950Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2125781Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2126138Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2127147Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2127676Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2127887Z super().__init__( 2025-10-10T02:02:57.2128223Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2129097Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2129419Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2130007Z next(self.gen) 2025-10-10T02:02:57.2130524Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2130886Z wait_for_engine_startup( 2025-10-10T02:02:57.2131071Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2131080Z 2025-10-10T02:02:57.2132944Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2133323Z "See root cause above. " 2025-10-10T02:02:57.2133869Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2134332Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2134337Z 2025-10-10T02:02:57.2134665Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2134919Z ___________ test_can_initialize_large_subset[InternLM2VEForCausalLM] ___________ 2025-10-10T02:02:57.2134922Z 2025-10-10T02:02:57.2135010Z model_arch = 'InternLM2VEForCausalLM' 2025-10-10T02:02:57.2135183Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a1bb0> 2025-10-10T02:02:57.2135187Z 2025-10-10T02:02:57.2139041Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2139512Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2139705Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2139931Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2140066Z  2025-10-10T02:02:57.2140423Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2140523Z  test. 2025-10-10T02:02:57.2140629Z  """ 2025-10-10T02:02:57.2140916Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2141367Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2141687Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2141692Z 2025-10-10T02:02:57.2141841Z models/test_initialization.py:131: 2025-10-10T02:02:57.2141979Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2142177Z utils.py:892: in wrapper 2025-10-10T02:02:57.2142366Z raise original_exception 2025-10-10T02:02:57.2142534Z utils.py:836: in wrapper 2025-10-10T02:02:57.2142651Z func(*args, **kwargs) 2025-10-10T02:02:57.2142824Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2142933Z LLM( 2025-10-10T02:02:57.2143419Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2143672Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2144222Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2144654Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2145151Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2145733Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2146096Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2147080Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2147465Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2147873Z super().__init__( 2025-10-10T02:02:57.2148261Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2149040Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2149280Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2149776Z next(self.gen) 2025-10-10T02:02:57.2150130Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2150529Z wait_for_engine_startup( 2025-10-10T02:02:57.2150637Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2150642Z 2025-10-10T02:02:57.2152684Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2153044Z "See root cause above. " 2025-10-10T02:02:57.2153416Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2153817Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2153825Z 2025-10-10T02:02:57.2154148Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2154391Z ___________ test_can_initialize_large_subset[Grok1ModelForCausalLM] ____________ 2025-10-10T02:02:57.2154395Z 2025-10-10T02:02:57.2154477Z model_arch = 'Grok1ModelForCausalLM' 2025-10-10T02:02:57.2154644Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f22124550a0> 2025-10-10T02:02:57.2154718Z 2025-10-10T02:02:57.2159934Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2160321Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2160610Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2160921Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2161010Z  2025-10-10T02:02:57.2161267Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2161362Z  test. 2025-10-10T02:02:57.2161476Z  """ 2025-10-10T02:02:57.2161764Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2162156Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2162416Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2162422Z 2025-10-10T02:02:57.2162560Z models/test_initialization.py:131: 2025-10-10T02:02:57.2162667Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2162778Z utils.py:892: in wrapper 2025-10-10T02:02:57.2162939Z raise original_exception 2025-10-10T02:02:57.2163042Z utils.py:836: in wrapper 2025-10-10T02:02:57.2163162Z func(*args, **kwargs) 2025-10-10T02:02:57.2163335Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2163890Z LLM( 2025-10-10T02:02:57.2164230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2165865Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2166240Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2167077Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2167427Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2168813Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2169172Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2170640Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2170975Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2171683Z super().__init__( 2025-10-10T02:02:57.2172015Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2173418Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2173647Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2174533Z next(self.gen) 2025-10-10T02:02:57.2174907Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2175386Z wait_for_engine_startup( 2025-10-10T02:02:57.2175486Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2175555Z 2025-10-10T02:02:57.2178002Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2178340Z "See root cause above. " 2025-10-10T02:02:57.2178701Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2179152Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2179157Z 2025-10-10T02:02:57.2179479Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2179729Z _____________ test_can_initialize_large_subset[SmolLM3ForCausalLM] _____________ 2025-10-10T02:02:57.2179736Z 2025-10-10T02:02:57.2179817Z model_arch = 'SmolLM3ForCausalLM' 2025-10-10T02:02:57.2180050Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a6090> 2025-10-10T02:02:57.2180054Z 2025-10-10T02:02:57.2185431Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2185825Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2186005Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2186226Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2186314Z  2025-10-10T02:02:57.2186569Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2186673Z  test. 2025-10-10T02:02:57.2186780Z  """ 2025-10-10T02:02:57.2187206Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2187561Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2187795Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2187800Z 2025-10-10T02:02:57.2187938Z models/test_initialization.py:131: 2025-10-10T02:02:57.2188044Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2188160Z utils.py:892: in wrapper 2025-10-10T02:02:57.2188321Z raise original_exception 2025-10-10T02:02:57.2188424Z utils.py:836: in wrapper 2025-10-10T02:02:57.2188541Z func(*args, **kwargs) 2025-10-10T02:02:57.2188720Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2188842Z LLM( 2025-10-10T02:02:57.2189158Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2190864Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2191231Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2192002Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2192342Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2193786Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2194140Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2195383Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2195735Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2196678Z super().__init__( 2025-10-10T02:02:57.2197138Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2198540Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2200011Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2200185Z next(self.gen) 2025-10-10T02:02:57.2200563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2200699Z wait_for_engine_startup( 2025-10-10T02:02:57.2200891Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2200944Z 2025-10-10T02:02:57.2203268Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2203600Z "See root cause above. " 2025-10-10T02:02:57.2203961Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2204337Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2204342Z 2025-10-10T02:02:57.2204672Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2204921Z _____________ test_can_initialize_large_subset[Mamba2ForCausalLM] ______________ 2025-10-10T02:02:57.2204925Z 2025-10-10T02:02:57.2205006Z model_arch = 'Mamba2ForCausalLM' 2025-10-10T02:02:57.2205180Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a17c0> 2025-10-10T02:02:57.2205186Z 2025-10-10T02:02:57.2210468Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2210786Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2210963Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2211188Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2211278Z  2025-10-10T02:02:57.2211535Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2211648Z  test. 2025-10-10T02:02:57.2211759Z  """ 2025-10-10T02:02:57.2212048Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2212386Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2212613Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2212618Z 2025-10-10T02:02:57.2212749Z models/test_initialization.py:131: 2025-10-10T02:02:57.2212845Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2212957Z utils.py:892: in wrapper 2025-10-10T02:02:57.2213217Z raise original_exception 2025-10-10T02:02:57.2213330Z utils.py:836: in wrapper 2025-10-10T02:02:57.2213455Z func(*args, **kwargs) 2025-10-10T02:02:57.2213663Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2214266Z LLM( 2025-10-10T02:02:57.2214648Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2216354Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2216724Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2217635Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2217976Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2219359Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2219821Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2221118Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2221476Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2222110Z super().__init__( 2025-10-10T02:02:57.2222445Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2223795Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2224033Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2225318Z next(self.gen) 2025-10-10T02:02:57.2225695Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2226009Z wait_for_engine_startup( 2025-10-10T02:02:57.2226120Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2226129Z 2025-10-10T02:02:57.2228444Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2228697Z "See root cause above. " 2025-10-10T02:02:57.2229197Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2229616Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2229624Z 2025-10-10T02:02:57.2229955Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2230206Z _____________ test_can_initialize_large_subset[FalconForCausalLM] ______________ 2025-10-10T02:02:57.2230212Z 2025-10-10T02:02:57.2230295Z model_arch = 'FalconForCausalLM' 2025-10-10T02:02:57.2230474Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a7650> 2025-10-10T02:02:57.2230478Z 2025-10-10T02:02:57.2235524Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2236018Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2236291Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2236516Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2236605Z  2025-10-10T02:02:57.2236861Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2236961Z  test. 2025-10-10T02:02:57.2237113Z  """ 2025-10-10T02:02:57.2237540Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2237893Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2238177Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2238185Z 2025-10-10T02:02:57.2238428Z models/test_initialization.py:131: 2025-10-10T02:02:57.2238534Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2238715Z utils.py:892: in wrapper 2025-10-10T02:02:57.2238917Z raise original_exception 2025-10-10T02:02:57.2239026Z utils.py:836: in wrapper 2025-10-10T02:02:57.2239261Z func(*args, **kwargs) 2025-10-10T02:02:57.2239442Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2239544Z LLM( 2025-10-10T02:02:57.2239943Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2241488Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2241850Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2242257Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2242604Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2244069Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2244432Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2245439Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2245793Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2246685Z super().__init__( 2025-10-10T02:02:57.2247034Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2248337Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2248568Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2249424Z next(self.gen) 2025-10-10T02:02:57.2249773Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2250242Z wait_for_engine_startup( 2025-10-10T02:02:57.2250354Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2250359Z 2025-10-10T02:02:57.2252887Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2253238Z "See root cause above. " 2025-10-10T02:02:57.2253749Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2254217Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2254223Z 2025-10-10T02:02:57.2254624Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2254995Z ___________ test_can_initialize_large_subset[DeepseekV32ForCausalLM] ___________ 2025-10-10T02:02:57.2255000Z 2025-10-10T02:02:57.2255100Z model_arch = 'DeepseekV32ForCausalLM' 2025-10-10T02:02:57.2255276Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212454c80> 2025-10-10T02:02:57.2255280Z 2025-10-10T02:02:57.2260339Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2260730Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2260953Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2261210Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2261303Z  2025-10-10T02:02:57.2261562Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2261658Z  test. 2025-10-10T02:02:57.2261768Z  """ 2025-10-10T02:02:57.2262062Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2262399Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2262630Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2262637Z 2025-10-10T02:02:57.2262772Z models/test_initialization.py:131: 2025-10-10T02:02:57.2262873Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2262987Z utils.py:892: in wrapper 2025-10-10T02:02:57.2263155Z raise original_exception 2025-10-10T02:02:57.2263265Z utils.py:836: in wrapper 2025-10-10T02:02:57.2263389Z func(*args, **kwargs) 2025-10-10T02:02:57.2263568Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2264164Z LLM( 2025-10-10T02:02:57.2264537Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2265937Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2266410Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2267207Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2267561Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2268879Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2269229Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2270763Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2271136Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2271657Z super().__init__( 2025-10-10T02:02:57.2272187Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2273439Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2273791Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2274693Z next(self.gen) 2025-10-10T02:02:57.2275078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2275454Z wait_for_engine_startup( 2025-10-10T02:02:57.2275554Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2275565Z 2025-10-10T02:02:57.2277939Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2278284Z "See root cause above. " 2025-10-10T02:02:57.2278775Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2279283Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2279288Z 2025-10-10T02:02:57.2279615Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2279862Z ______________ test_can_initialize_large_subset[SolarForCausalLM] ______________ 2025-10-10T02:02:57.2279866Z 2025-10-10T02:02:57.2279950Z model_arch = 'SolarForCausalLM' 2025-10-10T02:02:57.2280123Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a47d0> 2025-10-10T02:02:57.2280130Z 2025-10-10T02:02:57.2285393Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2285783Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2285983Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2286206Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2286292Z  2025-10-10T02:02:57.2286552Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2286647Z  test. 2025-10-10T02:02:57.2286758Z  """ 2025-10-10T02:02:57.2287145Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2287568Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2287803Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2287808Z 2025-10-10T02:02:57.2288022Z models/test_initialization.py:131: 2025-10-10T02:02:57.2288193Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2288316Z utils.py:892: in wrapper 2025-10-10T02:02:57.2288490Z raise original_exception 2025-10-10T02:02:57.2288595Z utils.py:836: in wrapper 2025-10-10T02:02:57.2288715Z func(*args, **kwargs) 2025-10-10T02:02:57.2288893Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2288991Z LLM( 2025-10-10T02:02:57.2289431Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2290970Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2291414Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2292150Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2292666Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2293873Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2294280Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2295852Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2296585Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2297082Z super().__init__( 2025-10-10T02:02:57.2297565Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2298558Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2298809Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2299501Z next(self.gen) 2025-10-10T02:02:57.2299860Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2300404Z wait_for_engine_startup( 2025-10-10T02:02:57.2300511Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2300521Z 2025-10-10T02:02:57.2303106Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2303411Z "See root cause above. " 2025-10-10T02:02:57.2303772Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2304156Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2304161Z 2025-10-10T02:02:57.2304485Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2304730Z __________________ test_can_initialize_large_subset[MiniCPMV] __________________ 2025-10-10T02:02:57.2304736Z 2025-10-10T02:02:57.2304807Z model_arch = 'MiniCPMV' 2025-10-10T02:02:57.2304983Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a3d70> 2025-10-10T02:02:57.2304987Z 2025-10-10T02:02:57.2310108Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2310591Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2310794Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2311027Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2311117Z  2025-10-10T02:02:57.2311380Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2311584Z  test. 2025-10-10T02:02:57.2311764Z  """ 2025-10-10T02:02:57.2312152Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2312617Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2312954Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2312959Z 2025-10-10T02:02:57.2313102Z models/test_initialization.py:131: 2025-10-10T02:02:57.2313203Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2313314Z utils.py:892: in wrapper 2025-10-10T02:02:57.2313483Z raise original_exception 2025-10-10T02:02:57.2313593Z utils.py:836: in wrapper 2025-10-10T02:02:57.2313727Z func(*args, **kwargs) 2025-10-10T02:02:57.2313901Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2314083Z LLM( 2025-10-10T02:02:57.2314563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2315476Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2315870Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2316210Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2316545Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2318055Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2318533Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2319372Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2321005Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2321184Z super().__init__( 2025-10-10T02:02:57.2321521Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2322233Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2322468Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2323889Z next(self.gen) 2025-10-10T02:02:57.2324331Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2324512Z wait_for_engine_startup( 2025-10-10T02:02:57.2324637Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2324644Z 2025-10-10T02:02:57.2327468Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2327875Z "See root cause above. " 2025-10-10T02:02:57.2328370Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2328763Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2328768Z 2025-10-10T02:02:57.2329164Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2329418Z ____________ test_can_initialize_large_subset[Ernie4_5ForCausalLM] _____________ 2025-10-10T02:02:57.2329424Z 2025-10-10T02:02:57.2329503Z model_arch = 'Ernie4_5ForCausalLM' 2025-10-10T02:02:57.2329676Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a5490> 2025-10-10T02:02:57.2329736Z 2025-10-10T02:02:57.2335335Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2335672Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2335862Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2336088Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2336183Z  2025-10-10T02:02:57.2336499Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2336602Z  test. 2025-10-10T02:02:57.2336751Z  """ 2025-10-10T02:02:57.2337045Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2337513Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2337743Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2337756Z 2025-10-10T02:02:57.2337992Z models/test_initialization.py:131: 2025-10-10T02:02:57.2338122Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2338243Z utils.py:892: in wrapper 2025-10-10T02:02:57.2338419Z raise original_exception 2025-10-10T02:02:57.2338533Z utils.py:836: in wrapper 2025-10-10T02:02:57.2338664Z func(*args, **kwargs) 2025-10-10T02:02:57.2338844Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2338978Z LLM( 2025-10-10T02:02:57.2339295Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2341105Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2341489Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2342104Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2342464Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2343832Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2344356Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2345517Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2345885Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2346483Z super().__init__( 2025-10-10T02:02:57.2346826Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2348240Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2348494Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2349335Z next(self.gen) 2025-10-10T02:02:57.2349717Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2349911Z wait_for_engine_startup( 2025-10-10T02:02:57.2350010Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2350091Z 2025-10-10T02:02:57.2352683Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2352939Z "See root cause above. " 2025-10-10T02:02:57.2353304Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2353753Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2353758Z 2025-10-10T02:02:57.2354120Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2354362Z __________________ test_can_initialize_large_subset[MiniCPMO] __________________ 2025-10-10T02:02:57.2354369Z 2025-10-10T02:02:57.2354453Z model_arch = 'MiniCPMO' 2025-10-10T02:02:57.2354632Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a2e10> 2025-10-10T02:02:57.2354636Z 2025-10-10T02:02:57.2359509Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2359837Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2360153Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2360405Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2360495Z  2025-10-10T02:02:57.2360752Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2360855Z  test. 2025-10-10T02:02:57.2360966Z  """ 2025-10-10T02:02:57.2361259Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2361595Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2361925Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2361930Z 2025-10-10T02:02:57.2362080Z models/test_initialization.py:131: 2025-10-10T02:02:57.2362177Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2362295Z utils.py:892: in wrapper 2025-10-10T02:02:57.2362528Z raise original_exception 2025-10-10T02:02:57.2362730Z utils.py:836: in wrapper 2025-10-10T02:02:57.2362881Z func(*args, **kwargs) 2025-10-10T02:02:57.2363068Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2363583Z LLM( 2025-10-10T02:02:57.2363933Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2365616Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2366005Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2366703Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2367052Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2368356Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2368830Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2370130Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2370494Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2371055Z super().__init__( 2025-10-10T02:02:57.2371525Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2372846Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2373184Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2373997Z next(self.gen) 2025-10-10T02:02:57.2374376Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2374641Z wait_for_engine_startup( 2025-10-10T02:02:57.2374746Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2374750Z 2025-10-10T02:02:57.2377053Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2377410Z "See root cause above. " 2025-10-10T02:02:57.2377784Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2378160Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2378165Z 2025-10-10T02:02:57.2378492Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2378761Z ___________ test_can_initialize_large_subset[DeepseekV3ForCausalLM] ____________ 2025-10-10T02:02:57.2378765Z 2025-10-10T02:02:57.2378850Z model_arch = 'DeepseekV3ForCausalLM' 2025-10-10T02:02:57.2379028Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a4a10> 2025-10-10T02:02:57.2379032Z 2025-10-10T02:02:57.2384221Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2384551Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2384742Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2384965Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2385051Z  2025-10-10T02:02:57.2385301Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2385400Z  test. 2025-10-10T02:02:57.2385506Z  """ 2025-10-10T02:02:57.2385798Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2386130Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2386434Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2386441Z 2025-10-10T02:02:57.2386576Z models/test_initialization.py:131: 2025-10-10T02:02:57.2386673Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2386783Z utils.py:892: in wrapper 2025-10-10T02:02:57.2387001Z raise original_exception 2025-10-10T02:02:57.2387109Z utils.py:836: in wrapper 2025-10-10T02:02:57.2387245Z func(*args, **kwargs) 2025-10-10T02:02:57.2387426Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2388090Z LLM( 2025-10-10T02:02:57.2388453Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2390151Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2390822Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2391213Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2391718Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2392645Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2393132Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2394101Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2394442Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2394918Z super().__init__( 2025-10-10T02:02:57.2395249Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2396669Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2396923Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2397528Z next(self.gen) 2025-10-10T02:02:57.2397908Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2398452Z wait_for_engine_startup( 2025-10-10T02:02:57.2398554Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2398559Z 2025-10-10T02:02:57.2401017Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2401356Z "See root cause above. " 2025-10-10T02:02:57.2401720Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2402115Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2402119Z 2025-10-10T02:02:57.2402458Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2402690Z __________________ test_can_initialize_large_subset[GteModel] __________________ 2025-10-10T02:02:57.2402699Z 2025-10-10T02:02:57.2402768Z model_arch = 'GteModel' 2025-10-10T02:02:57.2402941Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a2ea0> 2025-10-10T02:02:57.2403045Z 2025-10-10T02:02:57.2408262Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2408704Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2408911Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2409239Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2409329Z  2025-10-10T02:02:57.2409588Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2409761Z  test. 2025-10-10T02:02:57.2409908Z  """ 2025-10-10T02:02:57.2410210Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2410712Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2411009Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2411015Z 2025-10-10T02:02:57.2411158Z models/test_initialization.py:131: 2025-10-10T02:02:57.2411271Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2415661Z utils.py:892: in wrapper 2025-10-10T02:02:57.2415992Z raise original_exception 2025-10-10T02:02:57.2416111Z utils.py:836: in wrapper 2025-10-10T02:02:57.2416240Z func(*args, **kwargs) 2025-10-10T02:02:57.2416437Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2416614Z LLM( 2025-10-10T02:02:57.2416974Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2417186Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2417551Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2417825Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2418170Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2418391Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2418821Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2419109Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2419456Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2419634Z super().__init__( 2025-10-10T02:02:57.2419973Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2420208Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2420438Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2420606Z next(self.gen) 2025-10-10T02:02:57.2420957Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2421084Z wait_for_engine_startup( 2025-10-10T02:02:57.2421216Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2421315Z 2025-10-10T02:02:57.2422417Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2422772Z "See root cause above. " 2025-10-10T02:02:57.2423215Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2423598Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2423603Z 2025-10-10T02:02:57.2423940Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2424223Z ___ test_can_initialize_large_subset[LlavaOnevisionForConditionalGeneration] ___ 2025-10-10T02:02:57.2424229Z 2025-10-10T02:02:57.2424401Z model_arch = 'LlavaOnevisionForConditionalGeneration' 2025-10-10T02:02:57.2424607Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a53d0> 2025-10-10T02:02:57.2424612Z 2025-10-10T02:02:57.2429350Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2429676Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2429871Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2430099Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2430184Z  2025-10-10T02:02:57.2430441Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2430539Z  test. 2025-10-10T02:02:57.2430661Z  """ 2025-10-10T02:02:57.2430953Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2431292Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2431516Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2431520Z 2025-10-10T02:02:57.2431652Z models/test_initialization.py:131: 2025-10-10T02:02:57.2431752Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2431862Z utils.py:892: in wrapper 2025-10-10T02:02:57.2432027Z raise original_exception 2025-10-10T02:02:57.2432131Z utils.py:836: in wrapper 2025-10-10T02:02:57.2432252Z func(*args, **kwargs) 2025-10-10T02:02:57.2432434Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2432671Z LLM( 2025-10-10T02:02:57.2432994Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2434504Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2434897Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2435747Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2436094Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:100: in __init__ 2025-10-10T02:02:57.2437684Z self.processor = Processor(vllm_config=vllm_config, 2025-10-10T02:02:57.2438097Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/processor.py:56: in __init__ 2025-10-10T02:02:57.2438956Z self.mm_processor_cache = processor_cache_from_config( 2025-10-10T02:02:57.2439467Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:522: in processor_cache_from_config 2025-10-10T02:02:57.2440899Z if not _enable_processor_cache(model_config, mm_registry): 2025-10-10T02:02:57.2441282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/cache.py:488: in _enable_processor_cache 2025-10-10T02:02:57.2442298Z if not mm_registry.supports_multimodal_inputs(model_config): 2025-10-10T02:02:57.2442715Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:108: in supports_multimodal_inputs 2025-10-10T02:02:57.2444481Z info = self._create_processing_info(model_config, tokenizer=None) 2025-10-10T02:02:57.2444973Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:249: in _create_processing_info 2025-10-10T02:02:57.2445964Z model_cls = self._get_model_cls(model_config) 2025-10-10T02:02:57.2446332Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/multimodal/registry.py:231: in _get_model_cls 2025-10-10T02:02:57.2447530Z model_cls, _ = get_model_architecture(model_config) 2025-10-10T02:02:57.2447960Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:236: in get_model_architecture 2025-10-10T02:02:57.2448815Z model_arch = _get_model_architecture(model_config) 2025-10-10T02:02:57.2449258Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py:183: in _get_model_architecture 2025-10-10T02:02:57.2450807Z model_cls, arch = model_config.registry.resolve_model_cls( 2025-10-10T02:02:57.2451220Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:836: in resolve_model_cls 2025-10-10T02:02:57.2452094Z return self._raise_for_unsupported(architectures) 2025-10-10T02:02:57.2452202Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2452207Z 2025-10-10T02:02:57.2454046Z > raise ValueError( 2025-10-10T02:02:57.2454477Z f"Model architectures {architectures} failed " 2025-10-10T02:02:57.2454806Z "to be inspected. Please check the logs for more details.") 2025-10-10T02:02:57.2455308Z E ValueError: Model architectures ['LlavaOnevisionForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2455315Z 2025-10-10T02:02:57.2455696Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/model_executor/models/registry.py:614: ValueError 2025-10-10T02:02:57.2455936Z ______________ test_can_initialize_large_subset[FuyuForCausalLM] _______________ 2025-10-10T02:02:57.2455940Z 2025-10-10T02:02:57.2456023Z model_arch = 'FuyuForCausalLM' 2025-10-10T02:02:57.2456193Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a1fa0> 2025-10-10T02:02:57.2456196Z 2025-10-10T02:02:57.2461452Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2461777Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2462050Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2462276Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2462361Z  2025-10-10T02:02:57.2462619Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2462757Z  test. 2025-10-10T02:02:57.2462867Z  """ 2025-10-10T02:02:57.2463165Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2463581Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2463811Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2463818Z 2025-10-10T02:02:57.2464023Z models/test_initialization.py:131: 2025-10-10T02:02:57.2464257Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2464401Z utils.py:892: in wrapper 2025-10-10T02:02:57.2464570Z raise original_exception 2025-10-10T02:02:57.2464679Z utils.py:836: in wrapper 2025-10-10T02:02:57.2464799Z func(*args, **kwargs) 2025-10-10T02:02:57.2464980Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2465102Z LLM( 2025-10-10T02:02:57.2465424Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2467007Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2467444Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2468109Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2468478Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2469776Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2470168Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2471644Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2471992Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2472564Z super().__init__( 2025-10-10T02:02:57.2472961Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2474362Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2474600Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2475424Z next(self.gen) 2025-10-10T02:02:57.2475796Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2476008Z wait_for_engine_startup( 2025-10-10T02:02:57.2476107Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2476112Z 2025-10-10T02:02:57.2478486Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2478802Z "See root cause above. " 2025-10-10T02:02:57.2479267Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2479646Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2479696Z 2025-10-10T02:02:57.2480011Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2480256Z ________________ test_can_initialize_large_subset[AquilaModel] _________________ 2025-10-10T02:02:57.2480263Z 2025-10-10T02:02:57.2480335Z model_arch = 'AquilaModel' 2025-10-10T02:02:57.2480506Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a5dc0> 2025-10-10T02:02:57.2480510Z 2025-10-10T02:02:57.2485928Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2486524Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2486740Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2486973Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2487062Z  2025-10-10T02:02:57.2487312Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2487450Z  test. 2025-10-10T02:02:57.2487658Z  """ 2025-10-10T02:02:57.2487984Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2488434Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2488704Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2488711Z 2025-10-10T02:02:57.2488848Z models/test_initialization.py:131: 2025-10-10T02:02:57.2488960Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2489073Z utils.py:892: in wrapper 2025-10-10T02:02:57.2489236Z raise original_exception 2025-10-10T02:02:57.2489341Z utils.py:836: in wrapper 2025-10-10T02:02:57.2489465Z func(*args, **kwargs) 2025-10-10T02:02:57.2489637Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2489736Z LLM( 2025-10-10T02:02:57.2490195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2491183Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2491675Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2493114Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2493603Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2494390Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2494745Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2495766Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2496280Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2497120Z super().__init__( 2025-10-10T02:02:57.2497476Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2498879Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2499297Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2499941Z next(self.gen) 2025-10-10T02:02:57.2500490Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2500650Z wait_for_engine_startup( 2025-10-10T02:02:57.2500754Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2500765Z 2025-10-10T02:02:57.2503198Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2503531Z "See root cause above. " 2025-10-10T02:02:57.2504141Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2504550Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2504556Z 2025-10-10T02:02:57.2504890Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2505152Z __________ test_can_initialize_large_subset[StableLMEpochForCausalLM] __________ 2025-10-10T02:02:57.2505156Z 2025-10-10T02:02:57.2505257Z model_arch = 'StableLMEpochForCausalLM' 2025-10-10T02:02:57.2505428Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212454c80> 2025-10-10T02:02:57.2505434Z 2025-10-10T02:02:57.2510694Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2511013Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2511248Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2511656Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2511751Z  2025-10-10T02:02:57.2512019Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2512117Z  test. 2025-10-10T02:02:57.2512234Z  """ 2025-10-10T02:02:57.2512534Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2512972Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2513333Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2513341Z 2025-10-10T02:02:57.2513490Z models/test_initialization.py:131: 2025-10-10T02:02:57.2513593Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2513741Z utils.py:892: in wrapper 2025-10-10T02:02:57.2514055Z raise original_exception 2025-10-10T02:02:57.2514196Z utils.py:836: in wrapper 2025-10-10T02:02:57.2514317Z func(*args, **kwargs) 2025-10-10T02:02:57.2514580Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2514683Z LLM( 2025-10-10T02:02:57.2515012Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2516076Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2516453Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2517358Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2517699Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2519206Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2519568Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2521040Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2521524Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2522183Z super().__init__( 2025-10-10T02:02:57.2522521Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2523907Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2524149Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2525002Z next(self.gen) 2025-10-10T02:02:57.2525511Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2525845Z wait_for_engine_startup( 2025-10-10T02:02:57.2525985Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2525991Z 2025-10-10T02:02:57.2528335Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2528643Z "See root cause above. " 2025-10-10T02:02:57.2529103Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2529580Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2529585Z 2025-10-10T02:02:57.2529994Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2530319Z _______ test_can_initialize_large_subset[Gemma3ForConditionalGeneration] _______ 2025-10-10T02:02:57.2530324Z 2025-10-10T02:02:57.2530457Z model_arch = 'Gemma3ForConditionalGeneration' 2025-10-10T02:02:57.2530678Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa3cfda0> 2025-10-10T02:02:57.2530683Z 2025-10-10T02:02:57.2535686Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2536102Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2536327Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2536607Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2536712Z  2025-10-10T02:02:57.2537129Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2537253Z  test. 2025-10-10T02:02:57.2537399Z  """ 2025-10-10T02:02:57.2537764Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2538255Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2538532Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2538552Z 2025-10-10T02:02:57.2538729Z models/test_initialization.py:131: 2025-10-10T02:02:57.2538848Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2538991Z utils.py:892: in wrapper 2025-10-10T02:02:57.2539190Z raise original_exception 2025-10-10T02:02:57.2539324Z utils.py:836: in wrapper 2025-10-10T02:02:57.2539519Z func(*args, **kwargs) 2025-10-10T02:02:57.2539778Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2539909Z LLM( 2025-10-10T02:02:57.2540305Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2540847Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2541443Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2542155Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2542721Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2543834Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2544218Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2545558Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2546007Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2546652Z super().__init__( 2025-10-10T02:02:57.2547056Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2548317Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2548649Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2549585Z next(self.gen) 2025-10-10T02:02:57.2549951Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2550294Z wait_for_engine_startup( 2025-10-10T02:02:57.2550405Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2550410Z 2025-10-10T02:02:57.2552856Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2553238Z "See root cause above. " 2025-10-10T02:02:57.2553802Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2554271Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2554386Z 2025-10-10T02:02:57.2554806Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2555121Z ________ test_can_initialize_large_subset[GraniteMoeSharedForCausalLM] _________ 2025-10-10T02:02:57.2555126Z 2025-10-10T02:02:57.2555305Z model_arch = 'GraniteMoeSharedForCausalLM' 2025-10-10T02:02:57.2555509Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212454830> 2025-10-10T02:02:57.2555514Z 2025-10-10T02:02:57.2560640Z @pytest.mark.parametrize("model_arch", OTHER_MODEL_ARCH_LIST) 2025-10-10T02:02:57.2560985Z def test_can_initialize_large_subset(model_arch: str, 2025-10-10T02:02:57.2561172Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2561546Z  """Test initializing large subset of supported models 2025-10-10T02:02:57.2561714Z  2025-10-10T02:02:57.2562134Z  This test covers the complement of the tests covered in the "small subset" 2025-10-10T02:02:57.2562243Z  test. 2025-10-10T02:02:57.2562358Z  """ 2025-10-10T02:02:57.2562660Z if model_arch == "Lfm2ForCausalLM": 2025-10-10T02:02:57.2563229Z pytest.skip("Skipping until test supports V1-only models") 2025-10-10T02:02:57.2563523Z > can_initialize(model_arch, monkeypatch, HF_EXAMPLE_MODELS) 2025-10-10T02:02:57.2563538Z 2025-10-10T02:02:57.2563681Z models/test_initialization.py:131: 2025-10-10T02:02:57.2563793Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2563932Z utils.py:892: in wrapper 2025-10-10T02:02:57.2564289Z raise original_exception 2025-10-10T02:02:57.2564443Z utils.py:836: in wrapper 2025-10-10T02:02:57.2564582Z func(*args, **kwargs) 2025-10-10T02:02:57.2564798Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2564914Z LLM( 2025-10-10T02:02:57.2565299Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2566621Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2567003Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2567794Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2568137Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2569585Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2569951Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2571310Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2571692Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2572282Z super().__init__( 2025-10-10T02:02:57.2572635Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2574114Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2574520Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2575424Z next(self.gen) 2025-10-10T02:02:57.2575952Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2576292Z wait_for_engine_startup( 2025-10-10T02:02:57.2576403Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2576415Z 2025-10-10T02:02:57.2578935Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2579179Z "See root cause above. " 2025-10-10T02:02:57.2579756Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2580234Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2580240Z 2025-10-10T02:02:57.2580575Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2580840Z ________ test_implicit_converted_models[GemmaForSequenceClassification] ________ 2025-10-10T02:02:57.2580844Z 2025-10-10T02:02:57.2580955Z model_arch = 'GemmaForSequenceClassification' 2025-10-10T02:02:57.2581133Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a5a00> 2025-10-10T02:02:57.2581137Z 2025-10-10T02:02:57.2585630Z @pytest.mark.parametrize("model_arch", 2025-10-10T02:02:57.2585892Z AUTO_EXAMPLE_MODELS.get_supported_archs()) 2025-10-10T02:02:57.2586326Z def test_implicit_converted_models(model_arch: str, 2025-10-10T02:02:57.2586613Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2586868Z > can_initialize(model_arch, monkeypatch, AUTO_EXAMPLE_MODELS) 2025-10-10T02:02:57.2586875Z 2025-10-10T02:02:57.2587017Z models/test_initialization.py:138: 2025-10-10T02:02:57.2587120Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2587236Z utils.py:892: in wrapper 2025-10-10T02:02:57.2587432Z raise original_exception 2025-10-10T02:02:57.2587629Z utils.py:836: in wrapper 2025-10-10T02:02:57.2587994Z func(*args, **kwargs) 2025-10-10T02:02:57.2588181Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2588834Z LLM( 2025-10-10T02:02:57.2589175Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2590746Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2591132Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2591950Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2592292Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2593685Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2594046Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2595529Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2595918Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2596736Z super().__init__( 2025-10-10T02:02:57.2597083Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2598452Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2598690Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2599711Z next(self.gen) 2025-10-10T02:02:57.2600191Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2600579Z wait_for_engine_startup( 2025-10-10T02:02:57.2600844Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2600851Z 2025-10-10T02:02:57.2603103Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2603376Z "See root cause above. " 2025-10-10T02:02:57.2603945Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2604549Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2604555Z 2025-10-10T02:02:57.2604948Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2605279Z ________ test_implicit_converted_models[LlamaForSequenceClassification] ________ 2025-10-10T02:02:57.2605284Z 2025-10-10T02:02:57.2605416Z model_arch = 'LlamaForSequenceClassification' 2025-10-10T02:02:57.2605615Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2212455880> 2025-10-10T02:02:57.2605620Z 2025-10-10T02:02:57.2609581Z @pytest.mark.parametrize("model_arch", 2025-10-10T02:02:57.2609799Z AUTO_EXAMPLE_MODELS.get_supported_archs()) 2025-10-10T02:02:57.2610254Z def test_implicit_converted_models(model_arch: str, 2025-10-10T02:02:57.2610451Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2610783Z > can_initialize(model_arch, monkeypatch, AUTO_EXAMPLE_MODELS) 2025-10-10T02:02:57.2610801Z 2025-10-10T02:02:57.2610948Z models/test_initialization.py:138: 2025-10-10T02:02:57.2611049Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2611167Z utils.py:892: in wrapper 2025-10-10T02:02:57.2611338Z raise original_exception 2025-10-10T02:02:57.2611452Z utils.py:836: in wrapper 2025-10-10T02:02:57.2612170Z func(*args, **kwargs) 2025-10-10T02:02:57.2612356Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2612935Z LLM( 2025-10-10T02:02:57.2613264Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2615067Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2615527Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2616168Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2616517Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2617875Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2618307Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2619597Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2619952Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2620681Z super().__init__( 2025-10-10T02:02:57.2621025Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2622260Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2622536Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2623400Z next(self.gen) 2025-10-10T02:02:57.2623803Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2624318Z wait_for_engine_startup( 2025-10-10T02:02:57.2624420Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2624425Z 2025-10-10T02:02:57.2626860Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2627104Z "See root cause above. " 2025-10-10T02:02:57.2627577Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2628081Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2628094Z 2025-10-10T02:02:57.2628461Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2628729Z ________ test_implicit_converted_models[Qwen2ForSequenceClassification] ________ 2025-10-10T02:02:57.2628733Z 2025-10-10T02:02:57.2628838Z model_arch = 'Qwen2ForSequenceClassification' 2025-10-10T02:02:57.2629007Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa3cd040> 2025-10-10T02:02:57.2629010Z 2025-10-10T02:02:57.2633313Z @pytest.mark.parametrize("model_arch", 2025-10-10T02:02:57.2633665Z AUTO_EXAMPLE_MODELS.get_supported_archs()) 2025-10-10T02:02:57.2634125Z def test_implicit_converted_models(model_arch: str, 2025-10-10T02:02:57.2634457Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2634855Z > can_initialize(model_arch, monkeypatch, AUTO_EXAMPLE_MODELS) 2025-10-10T02:02:57.2634861Z 2025-10-10T02:02:57.2635096Z models/test_initialization.py:138: 2025-10-10T02:02:57.2635261Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2635455Z utils.py:892: in wrapper 2025-10-10T02:02:57.2635775Z raise original_exception 2025-10-10T02:02:57.2636060Z utils.py:836: in wrapper 2025-10-10T02:02:57.2636262Z func(*args, **kwargs) 2025-10-10T02:02:57.2636575Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2636746Z LLM( 2025-10-10T02:02:57.2637301Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2638473Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2639223Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2639838Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2640418Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2641412Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2642106Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2643299Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2643880Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2644323Z super().__init__( 2025-10-10T02:02:57.2644897Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2645919Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2646314Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2647093Z next(self.gen) 2025-10-10T02:02:57.2647707Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2647932Z wait_for_engine_startup( 2025-10-10T02:02:57.2648098Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2648104Z 2025-10-10T02:02:57.2650555Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2650982Z "See root cause above. " 2025-10-10T02:02:57.2651613Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2652256Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2652265Z 2025-10-10T02:02:57.2652607Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2652886Z ________ test_implicit_converted_models[Qwen3ForSequenceClassification] ________ 2025-10-10T02:02:57.2652890Z 2025-10-10T02:02:57.2652995Z model_arch = 'Qwen3ForSequenceClassification' 2025-10-10T02:02:57.2653176Z monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f23aa7a52e0> 2025-10-10T02:02:57.2653180Z 2025-10-10T02:02:57.2656458Z @pytest.mark.parametrize("model_arch", 2025-10-10T02:02:57.2656798Z AUTO_EXAMPLE_MODELS.get_supported_archs()) 2025-10-10T02:02:57.2657326Z def test_implicit_converted_models(model_arch: str, 2025-10-10T02:02:57.2657738Z monkeypatch: pytest.MonkeyPatch): 2025-10-10T02:02:57.2658142Z > can_initialize(model_arch, monkeypatch, AUTO_EXAMPLE_MODELS) 2025-10-10T02:02:57.2658148Z 2025-10-10T02:02:57.2658297Z models/test_initialization.py:138: 2025-10-10T02:02:57.2658402Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2658574Z utils.py:892: in wrapper 2025-10-10T02:02:57.2658829Z raise original_exception 2025-10-10T02:02:57.2659026Z utils.py:836: in wrapper 2025-10-10T02:02:57.2659230Z func(*args, **kwargs) 2025-10-10T02:02:57.2659543Z models/test_initialization.py:89: in can_initialize 2025-10-10T02:02:57.2659791Z LLM( 2025-10-10T02:02:57.2660347Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/entrypoints/llm.py:297: in __init__ 2025-10-10T02:02:57.2661755Z self.llm_engine = LLMEngine.from_engine_args( 2025-10-10T02:02:57.2662462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:177: in from_engine_args 2025-10-10T02:02:57.2662901Z return cls(vllm_config=vllm_config, 2025-10-10T02:02:57.2663456Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py:114: in __init__ 2025-10-10T02:02:57.2664326Z self.engine_core = EngineCoreClient.make_client( 2025-10-10T02:02:57.2664912Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:80: in make_client 2025-10-10T02:02:57.2665864Z return SyncMPClient(vllm_config, executor_class, log_stats) 2025-10-10T02:02:57.2666436Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:602: in __init__ 2025-10-10T02:02:57.2666969Z super().__init__( 2025-10-10T02:02:57.2667536Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/core_client.py:448: in __init__ 2025-10-10T02:02:57.2668537Z with launch_core_engines(vllm_config, executor_class, 2025-10-10T02:02:57.2668929Z /opt/conda/envs/py_3.12/lib/python3.12/contextlib.py:144: in __exit__ 2025-10-10T02:02:57.2669655Z next(self.gen) 2025-10-10T02:02:57.2670275Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:732: in launch_core_engines 2025-10-10T02:02:57.2670502Z wait_for_engine_startup( 2025-10-10T02:02:57.2670666Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2025-10-10T02:02:57.2670676Z 2025-10-10T02:02:57.2673189Z > raise RuntimeError("Engine core initialization failed. " 2025-10-10T02:02:57.2673453Z "See root cause above. " 2025-10-10T02:02:57.2673811Z f"Failed core proc(s): {finished}") 2025-10-10T02:02:57.2674192Z E RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2674202Z 2025-10-10T02:02:57.2674524Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/vllm/v1/engine/utils.py:785: RuntimeError 2025-10-10T02:02:57.2675280Z =============================== warnings summary =============================== 2025-10-10T02:02:57.2677768Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64 2025-10-10T02:02:57.2678954Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T02:02:57.2679152Z import pynvml # type: ignore[import] 2025-10-10T02:02:57.2679159Z 2025-10-10T02:02:57.2679634Z ../../../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305 2025-10-10T02:02:57.2680689Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable. 2025-10-10T02:02:57.2680858Z ref_error: type[Exception] = jsonschema.RefResolutionError, 2025-10-10T02:02:57.2680866Z 2025-10-10T02:02:57.2684709Z tests/models/test_initialization.py: 221 warnings 2025-10-10T02:02:57.2685248Z /var/lib/jenkins/workspace/vllm/tests/utils.py:829: DeprecationWarning: This process (pid=1343) is multi-threaded, use of fork() may lead to deadlocks in the child. 2025-10-10T02:02:57.2685325Z pid = os.fork() 2025-10-10T02:02:57.2685330Z 2025-10-10T02:02:57.2685507Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2025-10-10T02:02:57.2846924Z =========================== short test summary info ============================ 2025-10-10T02:02:57.2848561Z FAILED models/test_initialization.py::test_can_initialize_small_subset[LlavaForConditionalGeneration] - ValueError: Model architectures ['LlavaForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2850338Z FAILED models/test_initialization.py::test_can_initialize_small_subset[Llama4ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2851588Z FAILED models/test_initialization.py::test_can_initialize_small_subset[BertForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2852912Z FAILED models/test_initialization.py::test_can_initialize_small_subset[Gemma3nForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2854045Z FAILED models/test_initialization.py::test_can_initialize_small_subset[JinaVLForRanking] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2855059Z FAILED models/test_initialization.py::test_can_initialize_small_subset[InternVLChatModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2856147Z FAILED models/test_initialization.py::test_can_initialize_small_subset[InternLM2ForRewardModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2857006Z FAILED models/test_initialization.py::test_can_initialize_small_subset[TransformersForMultimodalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2857802Z FAILED models/test_initialization.py::test_can_initialize_small_subset[PrithviGeoSpatialMAE] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} 2025-10-10T02:02:57.2858484Z FAILED models/test_initialization.py::test_can_initialize_small_subset[UltravoxModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2859332Z FAILED models/test_initialization.py::test_can_initialize_small_subset[DeepSeekMTPModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2860017Z FAILED models/test_initialization.py::test_can_initialize_small_subset[XLMRobertaModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2860761Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Olmo2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2861426Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ErnieMTPModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2862207Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GptOssForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2862972Z FAILED models/test_initialization.py::test_can_initialize_large_subset[SmolVLMForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2863822Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LlavaNextForConditionalGeneration] - ValueError: Model architectures ['LlavaNextForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2864533Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Step3TextForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2865223Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BaichuanForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2866047Z FAILED models/test_initialization.py::test_can_initialize_large_subset[TarsierForConditionalGeneration] - ValueError: Model architectures ['TarsierForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2866753Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Eagle3LlamaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2867437Z FAILED models/test_initialization.py::test_can_initialize_large_subset[CohereForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2868131Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniMaxM1ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2868813Z FAILED models/test_initialization.py::test_can_initialize_large_subset[OlmoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2869500Z FAILED models/test_initialization.py::test_can_initialize_large_subset[HCXVisionForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2870190Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MistralForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2870922Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ArcticForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2871612Z FAILED models/test_initialization.py::test_can_initialize_large_subset[StableLmForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2874089Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Emu3ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2874817Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Exaone4ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2875606Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Glm4ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2876317Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BailingMoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2876993Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Gemma3TextModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2877680Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniCPMForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2878371Z FAILED models/test_initialization.py::test_can_initialize_large_subset[NemotronHForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2879197Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5_MoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2879887Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MambaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2880786Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LlavaNextVideoForConditionalGeneration] - ValueError: Model architectures ['LlavaNextVideoForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2881467Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GemmaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2882128Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Gemma2Model] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2882795Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5OmniModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2883522Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeHybridForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2884351Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Llama_Nemotron_Nano_VL] - RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method 2025-10-10T02:02:57.2885043Z FAILED models/test_initialization.py::test_can_initialize_large_subset[XverseForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2885831Z FAILED models/test_initialization.py::test_can_initialize_large_subset[InternS1ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2886621Z FAILED models/test_initialization.py::test_can_initialize_large_subset[PaliGemmaForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2887431Z FAILED models/test_initialization.py::test_can_initialize_large_subset[RForConditionalGeneration] - ValueError: Model architectures ['RForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2888117Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DeciLMForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2888800Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Gemma3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2889538Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Gemma3nForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2890279Z FAILED models/test_initialization.py::test_can_initialize_large_subset[QwenVLForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2890953Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LlamaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2891613Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MedusaModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2892350Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Glm4vForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2893045Z FAILED models/test_initialization.py::test_can_initialize_large_subset[InternLMForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2893720Z FAILED models/test_initialization.py::test_can_initialize_large_subset[NomicBertModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2894436Z FAILED models/test_initialization.py::test_can_initialize_large_subset[FalconMambaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2895186Z FAILED models/test_initialization.py::test_can_initialize_large_subset[AyaVisionForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2895922Z FAILED models/test_initialization.py::test_can_initialize_large_subset[FalconH1ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2896851Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MPTForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2897496Z FAILED models/test_initialization.py::test_can_initialize_large_subset[NVLM_D] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2898248Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GPT2LMHeadModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2899204Z FAILED models/test_initialization.py::test_can_initialize_large_subset[AriaForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2899881Z FAILED models/test_initialization.py::test_can_initialize_large_subset[OPTForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2900564Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BambaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2901343Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Cohere2VisionForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2902052Z FAILED models/test_initialization.py::test_can_initialize_large_subset[HunYuanMoEV1ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2902761Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Fairseq2LlamaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2903472Z FAILED models/test_initialization.py::test_can_initialize_large_subset[TransformersForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2904162Z FAILED models/test_initialization.py::test_can_initialize_large_subset[SkyworkR1VChatModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2904860Z FAILED models/test_initialization.py::test_can_initialize_large_subset[InternLM2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2905608Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5OmniForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2906351Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ChatGLMForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2907047Z FAILED models/test_initialization.py::test_can_initialize_large_subset[EagleDeepSeekMTPModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2907813Z FAILED models/test_initialization.py::test_can_initialize_large_subset[PersimmonForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2908583Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Eagle3Qwen2_5vlForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2909338Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2AudioForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2910053Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ExaoneForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2910757Z FAILED models/test_initialization.py::test_can_initialize_large_subset[RWForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2911443Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ApertusForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2912112Z FAILED models/test_initialization.py::test_can_initialize_large_subset[OlmoForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2912953Z FAILED models/test_initialization.py::test_can_initialize_large_subset[PixtralForConditionalGeneration] - ValueError: Model architectures ['PixtralForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2913647Z FAILED models/test_initialization.py::test_can_initialize_large_subset[JambaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2914372Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BailingMoeV2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2915056Z FAILED models/test_initialization.py::test_can_initialize_large_subset[TeleFLMForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2915714Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GteNewModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2916411Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DotsOCRForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2917163Z FAILED models/test_initialization.py::test_can_initialize_large_subset[RobertaForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2917846Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BloomForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2918578Z FAILED models/test_initialization.py::test_can_initialize_large_subset[KeyeForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2919482Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2_5_VLForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2920216Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2920958Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Idefics3ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2921684Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2922399Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Dots1ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2923077Z FAILED models/test_initialization.py::test_can_initialize_large_subset[JAISLMHeadModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2923763Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniCPM3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2924423Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MistralModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2925121Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DeepseekForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2925808Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GPTNeoXForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2926546Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2VLForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2927287Z FAILED models/test_initialization.py::test_can_initialize_large_subset[KeyeVL1_5ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2927966Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniMaxForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2928644Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Olmo3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2929334Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2MoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2930107Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GraniteSpeechForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2930840Z FAILED models/test_initialization.py::test_can_initialize_large_subset[AquilaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2931569Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Glm4MoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2932247Z FAILED models/test_initialization.py::test_can_initialize_large_subset[PhiMoEForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2933012Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GPT2ForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2933727Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ArceeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2934402Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Zamba2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2935146Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Step3VLForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2935842Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GPTBigCodeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2936523Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GlmForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2937223Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Starcoder2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2937907Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GraniteForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2938651Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Tarsier2ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2939414Z FAILED models/test_initialization.py::test_can_initialize_large_subset[XLMRobertaForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2940152Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GteNewForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2940842Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BaiChuanForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2941727Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Mistral3ForConditionalGeneration] - ValueError: Model architectures ['Mistral3ForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2942438Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LongCatFlashMTPModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2943224Z FAILED models/test_initialization.py::test_can_initialize_large_subset[InternVLForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2943901Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Phi3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2944645Z FAILED models/test_initialization.py::test_can_initialize_large_subset[RobertaModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2945357Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DeepseekV2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2946031Z FAILED models/test_initialization.py::test_can_initialize_large_subset[OrionForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2946703Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ModernBertModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2947375Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiMoForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2948076Z FAILED models/test_initialization.py::test_can_initialize_large_subset[TeleChat2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2948752Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiDashengLMModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2949403Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen2Model] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2950124Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BertForTokenClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2950817Z FAILED models/test_initialization.py::test_can_initialize_large_subset[InternLM3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2951510Z FAILED models/test_initialization.py::test_can_initialize_large_subset[EagleLlamaForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2952214Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LlamaForCausalLMEagle3] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2952918Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiMoMTPModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2953595Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GPTJForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2954395Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ModernBertForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2955068Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GLM4VForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2955860Z FAILED models/test_initialization.py::test_can_initialize_large_subset[ChameleonForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2956610Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2957285Z FAILED models/test_initialization.py::test_can_initialize_large_subset[PhiForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2958013Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Blip2ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2958700Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MixtralForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2959559Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5_VLMoeForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2960185Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Terratorch] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} 2025-10-10T02:02:57.2960891Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniMaxText01ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2961577Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Cohere2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2962278Z FAILED models/test_initialization.py::test_can_initialize_large_subset[EagleLlama4ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2962969Z FAILED models/test_initialization.py::test_can_initialize_large_subset[NemotronForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2963635Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DbrxForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2964426Z FAILED models/test_initialization.py::test_can_initialize_large_subset[JambaForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2965162Z FAILED models/test_initialization.py::test_can_initialize_large_subset[KimiVLForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2965838Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Ovis2_5] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2966516Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Phi4MMForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2967253Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Qwen3MoeForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2967979Z FAILED models/test_initialization.py::test_can_initialize_large_subset[RobertaForMaskedLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2968691Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LongcatFlashForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2969366Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Gemma2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2970091Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Phi4MultimodalForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2970845Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Glm4vMoeForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2971498Z FAILED models/test_initialization.py::test_can_initialize_large_subset[BertModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2972134Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GritLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2972840Z FAILED models/test_initialization.py::test_can_initialize_large_subset[InternLM2VEForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2973540Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Grok1ModelForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2974227Z FAILED models/test_initialization.py::test_can_initialize_large_subset[SmolLM3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2974901Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Mamba2ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2975580Z FAILED models/test_initialization.py::test_can_initialize_large_subset[FalconForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2976327Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DeepseekV32ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2977037Z FAILED models/test_initialization.py::test_can_initialize_large_subset[SolarForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2977688Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniCPMV] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2978401Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Ernie4_5ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2979109Z FAILED models/test_initialization.py::test_can_initialize_large_subset[MiniCPMO] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2979809Z FAILED models/test_initialization.py::test_can_initialize_large_subset[DeepseekV3ForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2980453Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GteModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2981343Z FAILED models/test_initialization.py::test_can_initialize_large_subset[LlavaOnevisionForConditionalGeneration] - ValueError: Model architectures ['LlavaOnevisionForConditionalGeneration'] failed to be inspected. Please check the logs for more details. 2025-10-10T02:02:57.2982037Z FAILED models/test_initialization.py::test_can_initialize_large_subset[FuyuForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2982698Z FAILED models/test_initialization.py::test_can_initialize_large_subset[AquilaModel] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2983411Z FAILED models/test_initialization.py::test_can_initialize_large_subset[StableLMEpochForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2984148Z FAILED models/test_initialization.py::test_can_initialize_large_subset[Gemma3ForConditionalGeneration] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2984875Z FAILED models/test_initialization.py::test_can_initialize_large_subset[GraniteMoeSharedForCausalLM] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2985605Z FAILED models/test_initialization.py::test_implicit_converted_models[GemmaForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2986338Z FAILED models/test_initialization.py::test_implicit_converted_models[LlamaForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2987066Z FAILED models/test_initialization.py::test_implicit_converted_models[Qwen2ForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2987848Z FAILED models/test_initialization.py::test_implicit_converted_models[Qwen3ForSequenceClassification] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1} 2025-10-10T02:02:57.2988243Z ===== 187 failed, 34 passed, 1 skipped, 223 warnings in 1545.41s (0:25:45) ===== 2025-10-10T02:02:58.9004920Z 2025-10-10 02:02:58,900 [INFO] cli.lib.core.vllm.lib: Finish running step: pytest -v -s models/test_initialization.py 2025-10-10T02:02:58.9006017Z 2025-10-10 02:02:58,900 [ERROR] cli.lib.core.vllm.lib: Failed tests: ['pytest -v -s models/test_transformers.py', 'pytest -v -s models/test_initialization.py'] 2025-10-10T02:02:58.9007013Z 2025-10-10 02:02:58,900 [INFO] cli.lib.core.vllm.vllm_test: Double check installed packages 2025-10-10T02:02:58.9014362Z 2025-10-10 02:02:58,901 [INFO] cli.lib.common.pip_helper: torch already exist with version: 2.10.0a0+git344e636 2025-10-10T02:02:58.9017688Z 2025-10-10 02:02:58,901 [INFO] cli.lib.common.pip_helper: xformers already exist with version: 0.0.33+5d4b92a5.d20251010 2025-10-10T02:02:58.9021392Z 2025-10-10 02:02:58,901 [INFO] cli.lib.common.pip_helper: torchvision already exist with version: 0.22.0a0+966da7e 2025-10-10T02:02:58.9025631Z 2025-10-10 02:02:58,902 [INFO] cli.lib.common.pip_helper: torchaudio already exist with version: 2.8.0a0+87ff22e 2025-10-10T02:02:58.9031369Z 2025-10-10 02:02:58,902 [INFO] cli.lib.common.pip_helper: vllm already exist with version: 0.11.0rc2.dev157+g0ad9951c4.d20251010 2025-10-10T02:02:58.9031967Z 2025-10-10 02:02:58,902 [INFO] cli.lib.core.vllm.vllm_test: Done. checked installed packages 2025-10-10T02:02:58.9032394Z Traceback (most recent call last): 2025-10-10T02:02:58.9032676Z File "", line 198, in _run_module_as_main 2025-10-10T02:02:58.9033064Z File "", line 88, in _run_code 2025-10-10T02:02:58.9033417Z File "/var/lib/jenkins/workspace/.ci/lumen_cli/cli/run.py", line 40, in 2025-10-10T02:02:58.9033856Z main() 2025-10-10T02:02:58.9034117Z File "/var/lib/jenkins/workspace/.ci/lumen_cli/cli/run.py", line 34, in main 2025-10-10T02:02:58.9034431Z args.func(args) 2025-10-10T02:02:58.9034891Z File "/var/lib/jenkins/workspace/.ci/lumen_cli/cli/lib/common/cli_helper.py", line 65, in 2025-10-10T02:02:58.9035319Z func=lambda args, cls=spec["runner"]: cls(args).run(), 2025-10-10T02:02:58.9035714Z ^^^^^^^^^^^^^^^ 2025-10-10T02:02:58.9036094Z File "/var/lib/jenkins/workspace/.ci/lumen_cli/cli/lib/core/vllm/vllm_test.py", line 125, in run 2025-10-10T02:02:58.9036572Z run_test_plan( 2025-10-10T02:02:58.9036920Z File "/var/lib/jenkins/workspace/.ci/lumen_cli/cli/lib/core/vllm/lib.py", line 262, in run_test_plan 2025-10-10T02:02:58.9037514Z raise RuntimeError(f"{len(failures)} pytest runs failed: {failures}") 2025-10-10T02:02:58.9038149Z RuntimeError: 2 pytest runs failed: ['pytest -v -s models/test_transformers.py', 'pytest -v -s models/test_initialization.py'] 2025-10-10T02:02:58.9283550Z + sccache_epilogue 2025-10-10T02:02:58.9283860Z + echo '::group::Sccache Compilation Log' 2025-10-10T02:02:58.9284562Z ##[group]Sccache Compilation Log 2025-10-10T02:02:58.9284922Z + echo '=================== sccache compilation log ===================' 2025-10-10T02:02:58.9285290Z =================== sccache compilation log =================== 2025-10-10T02:02:58.9285814Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-10-10T02:02:58.9718071Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-10-10T02:02:58.9718709Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-10-10T02:02:58.9719473Z + sccache --show-stats 2025-10-10T02:02:58.9799452Z Compile requests 0 2025-10-10T02:02:58.9800042Z Compile requests executed 0 2025-10-10T02:02:58.9800590Z Cache hits 0 2025-10-10T02:02:58.9801096Z Cache misses 0 2025-10-10T02:02:58.9801592Z Cache hits rate - 2025-10-10T02:02:58.9802371Z Cache timeouts 0 2025-10-10T02:02:58.9802848Z Cache read errors 0 2025-10-10T02:02:58.9803311Z Forced recaches 0 2025-10-10T02:02:58.9803788Z Cache write errors 0 2025-10-10T02:02:58.9804258Z Cache errors 0 2025-10-10T02:02:58.9804734Z Compilations 0 2025-10-10T02:02:58.9805216Z Compilation failures 0 2025-10-10T02:02:58.9805736Z Non-cacheable compilations 0 2025-10-10T02:02:58.9806253Z Non-cacheable calls 0 2025-10-10T02:02:58.9806726Z Non-compilation calls 0 2025-10-10T02:02:58.9807089Z Unsupported compiler calls 0 2025-10-10T02:02:58.9807323Z Average cache write 0.000 s 2025-10-10T02:02:58.9807631Z Average compiler 0.000 s 2025-10-10T02:02:58.9807861Z Average cache read hit 0.000 s 2025-10-10T02:02:58.9808092Z Failed distributed compilations 0 2025-10-10T02:02:58.9808420Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-10-10T02:02:58.9808750Z Version (client) 0.10.0 2025-10-10T02:02:58.9808980Z + sccache --stop-server 2025-10-10T02:02:58.9828783Z Stopping sccache server... 2025-10-10T02:02:58.9831562Z Compile requests 0 2025-10-10T02:02:58.9832114Z Compile requests executed 0 2025-10-10T02:02:58.9832591Z Cache hits 0 2025-10-10T02:02:58.9833025Z Cache misses 0 2025-10-10T02:02:58.9833308Z Cache hits rate - 2025-10-10T02:02:58.9833581Z Cache timeouts 0 2025-10-10T02:02:58.9833839Z Cache read errors 0 2025-10-10T02:02:58.9834093Z Forced recaches 0 2025-10-10T02:02:58.9834344Z Cache write errors 0 2025-10-10T02:02:58.9834592Z Cache errors 0 2025-10-10T02:02:58.9834851Z Compilations 0 2025-10-10T02:02:58.9835184Z Compilation failures 0 2025-10-10T02:02:58.9835662Z Non-cacheable compilations 0 2025-10-10T02:02:58.9836107Z Non-cacheable calls 0 2025-10-10T02:02:58.9836484Z Non-compilation calls 0 2025-10-10T02:02:58.9836878Z Unsupported compiler calls 0 2025-10-10T02:02:58.9837264Z Average cache write 0.000 s 2025-10-10T02:02:58.9837660Z Average compiler 0.000 s 2025-10-10T02:02:58.9837987Z Average cache read hit 0.000 s 2025-10-10T02:02:58.9838229Z Failed distributed compilations 0 2025-10-10T02:02:58.9838559Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-10-10T02:02:58.9838883Z Version (client) 0.10.0 2025-10-10T02:02:58.9839220Z + echo ::endgroup:: 2025-10-10T02:02:58.9839635Z ##[endgroup] 2025-10-10T02:02:58.9839837Z + cleanup_workspace 2025-10-10T02:02:58.9840197Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-10-10T02:02:58.9840739Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-10-10T02:02:58.9841187Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-10-10T02:02:58.9841510Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-10-10T02:02:58.9841884Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-10-10T02:02:58.9842468Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-10-10T02:02:58.9842795Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-10-10T02:03:00.0349751Z ##[error]Process completed with exit code 1. 2025-10-10T02:03:00.0404306Z Prepare all required actions 2025-10-10T02:03:00.0404660Z Getting action download info 2025-10-10T02:03:00.1875671Z ##[group]Run ./.github/actions/pytest-cache-upload 2025-10-10T02:03:00.1876028Z with: 2025-10-10T02:03:00.1876193Z cache_dir: .pytest_cache 2025-10-10T02:03:00.1876377Z shard: 1 2025-10-10T02:03:00.1876561Z sha: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T02:03:00.1876818Z test_config: vllm_basic_models_test 2025-10-10T02:03:00.1877106Z job_identifier: vllm-test_linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T02:03:00.1877375Z env: 2025-10-10T02:03:00.1877525Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:00.1877764Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:00.1878161Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:00.1878537Z ##[endgroup] 2025-10-10T02:03:00.1927339Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T02:03:00.1927561Z with: 2025-10-10T02:03:00.1927727Z shell: bash 2025-10-10T02:03:00.1927886Z timeout_minutes: 5 2025-10-10T02:03:00.1928055Z max_attempts: 5 2025-10-10T02:03:00.1928236Z retry_wait_seconds: 30 2025-10-10T02:03:00.1928476Z command: set -eu python3 -m pip install boto3==1.35.42 2025-10-10T02:03:00.1928745Z polling_interval_seconds: 1 2025-10-10T02:03:00.1928938Z warning_on_retry: true 2025-10-10T02:03:00.1929120Z continue_on_error: false 2025-10-10T02:03:00.1929294Z env: 2025-10-10T02:03:00.1929443Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:00.1929660Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:00.1930047Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:00.1930393Z ##[endgroup] 2025-10-10T02:03:00.5230353Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T02:03:01.5185961Z Collecting boto3==1.35.42 2025-10-10T02:03:01.5372029Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-10-10T02:03:01.5913490Z Collecting s3transfer<0.11.0,>=0.10.0 2025-10-10T02:03:01.5963285Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-10-10T02:03:01.6030766Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-10-10T02:03:02.6863306Z Collecting botocore<1.36.0,>=1.35.42 2025-10-10T02:03:02.6902441Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-10-10T02:03:02.8582270Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (2.8.1) 2025-10-10T02:03:02.8589604Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-10-10T02:03:03.0155958Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.15.0) 2025-10-10T02:03:03.0946609Z Installing collected packages: botocore, s3transfer, boto3 2025-10-10T02:03:03.6633223Z Successfully installed boto3-1.35.42 botocore-1.35.99 s3transfer-0.10.4 2025-10-10T02:03:04.2655871Z Command completed after 1 attempt(s). 2025-10-10T02:03:04.2722921Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2025-10-10T02:03:04.2723261Z python3 .github/scripts/pytest_cache.py \ 2025-10-10T02:03:04.2723520Z  --upload \ 2025-10-10T02:03:04.2723758Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2025-10-10T02:03:04.2724030Z  --pr_identifier $GITHUB_REF \ 2025-10-10T02:03:04.2724272Z  --job_identifier $JOB_IDENTIFIER \ 2025-10-10T02:03:04.2724520Z  --sha $SHA \ 2025-10-10T02:03:04.2724736Z  --test_config $TEST_CONFIG \ 2025-10-10T02:03:04.2725102Z  --shard $SHARD \ 2025-10-10T02:03:04.2725298Z  --repo $REPO \ 2025-10-10T02:03:04.2725500Z  --temp_dir $RUNNER_TEMP \ 2025-10-10T02:03:04.2737909Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:04.2738208Z env: 2025-10-10T02:03:04.2738369Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:04.2738623Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:04.2739124Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:04.2739503Z CACHE_DIR: .pytest_cache 2025-10-10T02:03:04.2739766Z JOB_IDENTIFIER: vllm-test_linux-jammy-cuda12.8-py3.12-gcc11 2025-10-10T02:03:04.2740068Z SHA: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T02:03:04.2740322Z TEST_CONFIG: vllm_basic_models_test 2025-10-10T02:03:04.2740530Z SHARD: 1 2025-10-10T02:03:04.2740687Z REPO: pytorch/pytorch 2025-10-10T02:03:04.2740862Z ##[endgroup] 2025-10-10T02:03:04.4056817Z PR identifier for `refs/heads/main` is `96e092540d6b3c4076e3d2bc6f1f9013` 2025-10-10T02:03:04.4058605Z Uploading cache with args Namespace(upload=True, download=False, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', pr_identifier='refs/heads/main', job_identifier='vllm-test_linux-jammy-cuda12.8-py3.12-gcc11', sha='344e6365a0068c2d2847fcec0c55dd53291d475e', test_config='vllm_basic_models_test', shard='1', repo='pytorch/pytorch', temp_dir='/home/ec2-user/actions-runner/_work/_temp', bucket=None) 2025-10-10T02:03:04.4060469Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2025-10-10T02:03:04.4248333Z ##[group]Run cat test/**/*_toprint.log || true 2025-10-10T02:03:04.4248669Z cat test/**/*_toprint.log || true 2025-10-10T02:03:04.4256460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:04.4256736Z env: 2025-10-10T02:03:04.4256911Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:04.4257172Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:04.4257580Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:04.4257946Z ##[endgroup] 2025-10-10T02:03:04.4354484Z cat: 'test/**/*_toprint.log': No such file or directory 2025-10-10T02:03:04.4385145Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-10-10T02:03:04.4385457Z kill "$MONITOR_SCRIPT_PID" 2025-10-10T02:03:04.4392663Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:04.4392936Z env: 2025-10-10T02:03:04.4393095Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:04.4393340Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:04.4393738Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:04.4394113Z MONITOR_SCRIPT_PID: 60313 2025-10-10T02:03:04.4394308Z ##[endgroup] 2025-10-10T02:03:04.4511209Z Prepare all required actions 2025-10-10T02:03:04.4511576Z Getting action download info 2025-10-10T02:03:04.6102240Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-10-10T02:03:04.8269063Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-10-10T02:03:05.2104736Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-10-10T02:03:05.2104995Z with: 2025-10-10T02:03:05.2105322Z file-suffix: test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T02:03:05.2105721Z s3-bucket: gha-artifacts 2025-10-10T02:03:05.2105908Z env: 2025-10-10T02:03:05.2106063Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.2106302Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.2106703Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.2107063Z ##[endgroup] 2025-10-10T02:03:05.2130660Z ##[group]Run # Remove any previous test jsons if they exist 2025-10-10T02:03:05.2131122Z # Remove any previous test jsons if they exist 2025-10-10T02:03:05.2131390Z rm -f test-jsons-*.zip 2025-10-10T02:03:05.2131691Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-10-10T02:03:05.2139791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:05.2140065Z env: 2025-10-10T02:03:05.2140220Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.2140584Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.2140984Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.2141525Z FILE_SUFFIX: test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T02:03:05.2141913Z ##[endgroup] 2025-10-10T02:03:05.2274269Z zip warning: name not matched: test/test-reports 2025-10-10T02:03:05.2275311Z zip warning: zip file empty 2025-10-10T02:03:05.2302913Z ##[group]Run # Remove any previous test reports if they exist 2025-10-10T02:03:05.2303263Z # Remove any previous test reports if they exist 2025-10-10T02:03:05.2303542Z rm -f test-reports-*.zip 2025-10-10T02:03:05.2303873Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-10-10T02:03:05.2310909Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:05.2311195Z env: 2025-10-10T02:03:05.2311353Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.2311602Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.2312003Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.2312528Z FILE_SUFFIX: test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T02:03:05.2312909Z ##[endgroup] 2025-10-10T02:03:05.2363583Z zip warning: name not matched: test/test-reports 2025-10-10T02:03:05.2364318Z zip warning: zip file empty 2025-10-10T02:03:05.2396978Z ##[group]Run # Remove any previous usage logs if they exist 2025-10-10T02:03:05.2397343Z # Remove any previous usage logs if they exist 2025-10-10T02:03:05.2397603Z rm -f logs-*.zip 2025-10-10T02:03:05.2397851Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-10-10T02:03:05.2398197Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-10-10T02:03:05.2405203Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:05.2405476Z env: 2025-10-10T02:03:05.2405631Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.2405859Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.2406251Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.2406771Z FILE_SUFFIX: test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T02:03:05.2407334Z ##[endgroup] 2025-10-10T02:03:05.2511831Z adding: usage_log.txt (deflated 96%) 2025-10-10T02:03:05.2525084Z zip warning: name not matched: test/test-reports 2025-10-10T02:03:05.2525324Z 2025-10-10T02:03:05.2525712Z zip error: Nothing to do! (logs-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip) 2025-10-10T02:03:05.2557227Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-10-10T02:03:05.2557610Z # Remove any previous debugging artifacts if they exist 2025-10-10T02:03:05.2557903Z rm -f debug-*.zip 2025-10-10T02:03:05.2558113Z if [ -d 'test/debug' ]; then 2025-10-10T02:03:05.2558375Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-10-10T02:03:05.2558616Z fi 2025-10-10T02:03:05.2565716Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:05.2565998Z env: 2025-10-10T02:03:05.2566159Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.2566412Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.2566811Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.2567453Z FILE_SUFFIX: test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898 2025-10-10T02:03:05.2567843Z ##[endgroup] 2025-10-10T02:03:05.2646505Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T02:03:05.2646743Z with: 2025-10-10T02:03:05.2646901Z s3-bucket: gha-artifacts 2025-10-10T02:03:05.2647137Z s3-prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:05.2647524Z retention-days: 14 2025-10-10T02:03:05.2647707Z if-no-files-found: warn 2025-10-10T02:03:05.2647901Z path: test-jsons-*.zip 2025-10-10T02:03:05.2648079Z name: artifact 2025-10-10T02:03:05.2648244Z region: us-east-1 2025-10-10T02:03:05.2648403Z env: 2025-10-10T02:03:05.2648557Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.2648790Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.2649195Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.2649551Z ##[endgroup] 2025-10-10T02:03:05.5585935Z NOTE: s3-prefix specified, ignoring name parameter 2025-10-10T02:03:05.5586608Z With the provided path, there will be 1 file uploaded 2025-10-10T02:03:05.5587217Z Uploading to s3 prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:05.5654460Z Starting upload of test-jsons-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:05.6719549Z Finished upload of test-jsons-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:05.6948547Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T02:03:05.6948785Z with: 2025-10-10T02:03:05.6948961Z s3-bucket: gha-artifacts 2025-10-10T02:03:05.6949207Z s3-prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:05.6949466Z retention-days: 14 2025-10-10T02:03:05.6949665Z if-no-files-found: error 2025-10-10T02:03:05.6949866Z path: test-reports-*.zip 2025-10-10T02:03:05.6950052Z name: artifact 2025-10-10T02:03:05.6950226Z region: us-east-1 2025-10-10T02:03:05.6950385Z env: 2025-10-10T02:03:05.6950533Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:05.6950785Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:05.6951183Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:05.6951532Z ##[endgroup] 2025-10-10T02:03:06.0221106Z NOTE: s3-prefix specified, ignoring name parameter 2025-10-10T02:03:06.0221812Z With the provided path, there will be 1 file uploaded 2025-10-10T02:03:06.0222314Z Uploading to s3 prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:06.0288377Z Starting upload of test-reports-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:06.1406063Z Finished upload of test-reports-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:06.1696907Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T02:03:06.1697163Z with: 2025-10-10T02:03:06.1697324Z s3-bucket: gha-artifacts 2025-10-10T02:03:06.1697563Z s3-prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:06.1697819Z retention-days: 14 2025-10-10T02:03:06.1698001Z if-no-files-found: ignore 2025-10-10T02:03:06.1698187Z path: logs-*.zip 2025-10-10T02:03:06.1698357Z name: artifact 2025-10-10T02:03:06.1698519Z region: us-east-1 2025-10-10T02:03:06.1698698Z env: 2025-10-10T02:03:06.1698846Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:06.1699084Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:06.1699487Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:06.1699841Z ##[endgroup] 2025-10-10T02:03:06.4661319Z NOTE: s3-prefix specified, ignoring name parameter 2025-10-10T02:03:06.4661770Z With the provided path, there will be 1 file uploaded 2025-10-10T02:03:06.4662234Z Uploading to s3 prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:06.4728774Z Starting upload of logs-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:06.6189734Z Finished upload of logs-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:06.6430114Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T02:03:06.6430350Z with: 2025-10-10T02:03:06.6430515Z s3-bucket: gha-artifacts 2025-10-10T02:03:06.6430872Z s3-prefix: pytorch/pytorch/18392306035/1/artifact 2025-10-10T02:03:06.6431127Z retention-days: 14 2025-10-10T02:03:06.6431303Z if-no-files-found: ignore 2025-10-10T02:03:06.6431495Z path: debug-*.zip 2025-10-10T02:03:06.6431663Z name: artifact 2025-10-10T02:03:06.6431833Z region: us-east-1 2025-10-10T02:03:06.6431988Z env: 2025-10-10T02:03:06.6432144Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:06.6432418Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:06.6432827Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:06.6433189Z ##[endgroup] 2025-10-10T02:03:06.9308462Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-10-10T02:03:06.9544773Z ##[group]Run # shellcheck disable=SC2156 2025-10-10T02:03:06.9545092Z # shellcheck disable=SC2156 2025-10-10T02:03:06.9545513Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-10-10T02:03:06.9553543Z shell: /usr/bin/bash -e {0} 2025-10-10T02:03:06.9553751Z env: 2025-10-10T02:03:06.9553921Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:06.9554165Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:06.9554571Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:06.9554935Z ##[endgroup] 2025-10-10T02:03:07.3559849Z ##[group]Run seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a 2025-10-10T02:03:07.3560284Z with: 2025-10-10T02:03:07.3560595Z name: coredumps-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu 2025-10-10T02:03:07.3560955Z retention-days: 14 2025-10-10T02:03:07.3561142Z if-no-files-found: ignore 2025-10-10T02:03:07.3561352Z path: ./**/core.[1-9]* 2025-10-10T02:03:07.3561547Z s3-bucket: gha-artifacts 2025-10-10T02:03:07.3561754Z region: us-east-1 2025-10-10T02:03:07.3561916Z env: 2025-10-10T02:03:07.3562076Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:07.3562318Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:07.3562714Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:07.3563064Z ##[endgroup] 2025-10-10T02:03:17.9886654Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2025-10-10T02:03:18.0202773Z Prepare all required actions 2025-10-10T02:03:18.0203104Z Getting action download info 2025-10-10T02:03:18.1390213Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-10-10T02:03:18.1390489Z with: 2025-10-10T02:03:18.1390646Z job_id: 52408504898 2025-10-10T02:03:18.1390995Z job_name: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T02:03:18.1391386Z workflow_name: vllm-test 2025-10-10T02:03:18.1391583Z workflow_run_id: 18392306035 2025-10-10T02:03:18.1391775Z workflow_attempt: 1 2025-10-10T02:03:18.1391954Z env: 2025-10-10T02:03:18.1392119Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:18.1392361Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:18.1392761Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:18.1393136Z ##[endgroup] 2025-10-10T02:03:18.1424850Z ##[group]Run echo "workflow_id: 18392306035" 2025-10-10T02:03:18.1425146Z echo "workflow_id: 18392306035" 2025-10-10T02:03:18.1425390Z echo "workflow_attempt: 1" 2025-10-10T02:03:18.1425614Z echo "workflow_Name: vllm-test" 2025-10-10T02:03:18.1425959Z echo "job_id: 52408504898" 2025-10-10T02:03:18.1426370Z echo "job_name: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu)" 2025-10-10T02:03:18.1426807Z echo "artifact_prefix: " 2025-10-10T02:03:18.1427018Z python3 --version 2025-10-10T02:03:18.1434732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:18.1435105Z env: 2025-10-10T02:03:18.1435263Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:18.1435499Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:18.1435899Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:18.1436270Z ##[endgroup] 2025-10-10T02:03:18.1463934Z workflow_id: 18392306035 2025-10-10T02:03:18.1464201Z workflow_attempt: 1 2025-10-10T02:03:18.1464458Z workflow_Name: vllm-test 2025-10-10T02:03:18.1464684Z job_id: 52408504898 2025-10-10T02:03:18.1465107Z job_name: ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu) 2025-10-10T02:03:18.1465583Z artifact_prefix: 2025-10-10T02:03:18.1479717Z Python 3.9.23 2025-10-10T02:03:18.1529842Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T02:03:18.1530061Z with: 2025-10-10T02:03:18.1530202Z shell: bash 2025-10-10T02:03:18.1530363Z timeout_minutes: 5 2025-10-10T02:03:18.1530535Z max_attempts: 5 2025-10-10T02:03:18.1530704Z retry_wait_seconds: 30 2025-10-10T02:03:18.1531096Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-10-10T02:03:18.1531518Z polling_interval_seconds: 1 2025-10-10T02:03:18.1531721Z warning_on_retry: true 2025-10-10T02:03:18.1531906Z continue_on_error: false 2025-10-10T02:03:18.1532097Z env: 2025-10-10T02:03:18.1532253Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:18.1532486Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:18.1532894Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:18.1533255Z ##[endgroup] 2025-10-10T02:03:18.4660424Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T02:03:18.5401146Z Collecting python-dateutil==2.8.2 2025-10-10T02:03:18.5651597Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-10-10T02:03:18.5787421Z Requirement already satisfied: boto3==1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.42) 2025-10-10T02:03:19.0489763Z Collecting pandas==2.1.3 2025-10-10T02:03:19.0556255Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-10-10T02:03:19.2942507Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-10-10T02:03:19.2957544Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-10-10T02:03:19.2997529Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-10-10T02:03:19.3002483Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-10-10T02:03:19.3005641Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-10-10T02:03:20.1084427Z Collecting numpy<2,>=1.22.4 2025-10-10T02:03:20.1142417Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-10-10T02:03:20.2932601Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-10-10T02:03:20.3220399Z Collecting tzdata>=2022.1 2025-10-10T02:03:20.3264706Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-10-10T02:03:20.3376925Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-10-10T02:03:20.3380413Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-10-10T02:03:20.3432412Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-10-10T02:03:20.3537630Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-10-10T02:03:20.3627295Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-10-10T02:03:20.3630731Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.15.0) 2025-10-10T02:03:20.5140805Z Installing collected packages: python-dateutil, tzdata, numpy, pandas 2025-10-10T02:03:24.8187895Z Successfully installed numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-10-10T02:03:25.2281680Z Command completed after 1 attempt(s). 2025-10-10T02:03:25.2357006Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-10-10T02:03:25.2357497Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-10-10T02:03:25.2357871Z  --workflow-run-id "18392306035" \ 2025-10-10T02:03:25.2358121Z  --workflow-name "vllm-test" \ 2025-10-10T02:03:25.2358373Z  --workflow-run-attempt "1" \ 2025-10-10T02:03:25.2358601Z  --job-id "52408504898" \ 2025-10-10T02:03:25.2359133Z  --job-name "ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu)" \ 2025-10-10T02:03:25.2359555Z  --local-path "" \ 2025-10-10T02:03:25.2359777Z  --artifact-prefix "" 2025-10-10T02:03:25.2367632Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:25.2367929Z env: 2025-10-10T02:03:25.2368091Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:25.2368343Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:25.2368745Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:25.2369098Z ##[endgroup] 2025-10-10T02:03:26.6158765Z repo: pytorch/pytorch 2025-10-10T02:03:26.6159222Z Search for test log in s3 bucket: ossci-utilization 2025-10-10T02:03:26.6159827Z Downloading logs-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:26.6161031Z extracting usage_log.txt from zip file logs-test-vllm_basic_models_test-1-1-linux.g6.4xlarge.experimental.nvidia.gpu_52408504898.zip 2025-10-10T02:03:26.6161569Z Converted Log Model: UtilizationMetadata: 2025-10-10T02:03:26.6162591Z UtilizationMetadata(level='metadata', workflow_id='18392306035', job_id='52408504898', workflow_name='vllm-test', job_name='ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1760059854, gpu_count=1, cpu_count=16, gpu_type='pynvml', error=None) 2025-10-10T02:03:26.6163818Z [Db Segments] detected pytest cmd: 9, generated segments: 9 2025-10-10T02:03:26.6164261Z [db model] Peek db timeseries 2025-10-10T02:03:26.6164472Z :{ 2025-10-10T02:03:26.6164621Z "created_at": 1760061806, 2025-10-10T02:03:26.6164815Z "type": "utilization", 2025-10-10T02:03:26.6165005Z "tags": [ 2025-10-10T02:03:26.6165150Z "record" 2025-10-10T02:03:26.6165301Z ], 2025-10-10T02:03:26.6165451Z "time_stamp": 1760059854, 2025-10-10T02:03:26.6165660Z "repo": "pytorch/pytorch", 2025-10-10T02:03:26.6165855Z "workflow_id": 18392306035, 2025-10-10T02:03:26.6166050Z "run_attempt": 1, 2025-10-10T02:03:26.6166221Z "job_id": 52408504898, 2025-10-10T02:03:26.6166571Z "workflow_name": "vllm-test", 2025-10-10T02:03:26.6166947Z "job_name": "ci-vllm-test / test (vllm_basic_models_test, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu)", 2025-10-10T02:03:26.6167328Z "json_data": "{}" 2025-10-10T02:03:26.6167505Z } 2025-10-10T02:03:26.6167850Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/18392306035/1/52408504898/metadata 2025-10-10T02:03:26.6168601Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/18392306035/1/52408504898/metadata 2025-10-10T02:03:26.6169250Z Writing 381 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/18392306035/1/52408504898/time_series 2025-10-10T02:03:26.6169931Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/18392306035/1/52408504898/time_series 2025-10-10T02:03:26.7014766Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-10-10T02:03:26.7015116Z with: 2025-10-10T02:03:26.7015280Z env: 2025-10-10T02:03:26.7015431Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:26.7015695Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:26.7016104Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:26.7016457Z ##[endgroup] 2025-10-10T02:03:26.7035684Z ##[group]Run set -eou pipefail 2025-10-10T02:03:26.7035925Z set -eou pipefail 2025-10-10T02:03:26.7036144Z  2025-10-10T02:03:26.7036409Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-10-10T02:03:26.7036731Z for _ in $(seq 1440); do 2025-10-10T02:03:26.7036967Z  # Break if no ssh session exists anymore 2025-10-10T02:03:26.7037215Z  if [ "$(who)" = "" ]; then 2025-10-10T02:03:26.7037497Z  break 2025-10-10T02:03:26.7037692Z  fi 2025-10-10T02:03:26.7037856Z  echo "." 2025-10-10T02:03:26.7038022Z  sleep 5 2025-10-10T02:03:26.7038192Z done 2025-10-10T02:03:26.7046250Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:26.7046531Z env: 2025-10-10T02:03:26.7046691Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:26.7046937Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:26.7047337Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:26.7047702Z ##[endgroup] 2025-10-10T02:03:26.7076846Z Holding runner for 2 hours until all ssh sessions have logged out 2025-10-10T02:03:26.7174879Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T02:03:26.7175291Z # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T02:03:26.7175592Z # shellcheck disable=SC2046 2025-10-10T02:03:26.7175841Z docker stop $(docker ps -q) || true 2025-10-10T02:03:26.7176083Z # Prune all of the docker images 2025-10-10T02:03:26.7176315Z docker system prune -af 2025-10-10T02:03:26.7183180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:26.7183455Z env: 2025-10-10T02:03:26.7183606Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:26.7183847Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:26.7184242Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:26.7184600Z ##[endgroup] 2025-10-10T02:03:38.8178516Z 3e581cf95cad 2025-10-10T02:03:46.6289564Z Deleted Containers: 2025-10-10T02:03:46.6290021Z 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:46.6290343Z 2025-10-10T02:03:58.1297115Z Deleted Images: 2025-10-10T02:03:58.1298472Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T02:03:58.1300341Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:dbf8617838c93f0ebd67b8b8bb938f5947e45ccc171ead9c6da61e625c32803f 2025-10-10T02:03:58.1301382Z deleted: sha256:fb0d053588b3e7d110638a9f678211e76aa4515ed1bad57389e38e2ab079e7b1 2025-10-10T02:03:58.1301847Z deleted: sha256:839691e084b3ec6bcbfe66d9e2f381fd843ef9f96565a69ff80aa86506df4ab2 2025-10-10T02:03:58.1302294Z deleted: sha256:14b10d5599eb456683f22fba693584a0ef542f8590cc214f13f622cdde3ffda4 2025-10-10T02:03:58.1302746Z deleted: sha256:4571fb4b2d2c700ea9adf9affdc419a0ae177bb9383a07181383b9d17371480e 2025-10-10T02:03:58.1303335Z deleted: sha256:d70dcb9b01054c38ce00200391740bcebd3a1cd72afe53327b81536008898606 2025-10-10T02:03:58.1303769Z deleted: sha256:8b8e5f7b43e9465220aafb7425efb40f3bfc014cc63b0364f673a990df3aa535 2025-10-10T02:03:58.1304229Z deleted: sha256:7d48d66ee4d478c78c3a6adadf587cf95bd748322ce32fff444f5d33ef2e0684 2025-10-10T02:03:58.1304896Z deleted: sha256:e4cd8739685cdd499ba08e895a29d365b8cb699f94cbb6b67e4e3619050a6dcc 2025-10-10T02:03:58.1305346Z deleted: sha256:559b9e576cde4002024cdd33e43847270549d69ec51a9446714cc1116826c5c4 2025-10-10T02:03:58.1305793Z deleted: sha256:d35159fdcc78c1037a6857000f6d1f7369c8dbe2b4e525aa0d66ee6ab394a829 2025-10-10T02:03:58.1306232Z deleted: sha256:1c1e45032bdc2f01b125fa0145c205376fce6aa2f8c2fbe5d7d286d8eb8efde6 2025-10-10T02:03:58.1306667Z deleted: sha256:e731b5134da5b9c47ce1696070037429ca15f364df4814a33bed968d293675ad 2025-10-10T02:03:58.1307107Z deleted: sha256:76e8028e50dc41cd6016284c7f7ab521b5b26a1f040406e7acfdd45d783aec82 2025-10-10T02:03:58.1307559Z deleted: sha256:da7adf94e8e4e4a5f8d1743254a6f6940542d2be33bf27fbf5390e8327ff9ec8 2025-10-10T02:03:58.1307999Z deleted: sha256:9f739ff4c0b7b0b7e859d014c1429bf8aeae2fbc74c4f95a729db4e18b6016f1 2025-10-10T02:03:58.1308440Z deleted: sha256:da101355a6a1035e5d8e120d6fcf20d6134887b145b5ba47cefc89942896b034 2025-10-10T02:03:58.1308885Z deleted: sha256:ec55a4fb03fa5b68218c181b9f8b5b48718b3b7a9f2688bc2a2ddebbc7e702d2 2025-10-10T02:03:58.1309614Z deleted: sha256:fc582e3da7f990556f9a46cc65e276a17e7b968253b500622a9fc411601644c1 2025-10-10T02:03:58.1310169Z deleted: sha256:3877d9b3300249f1fc15fa5be7f75e7368e3c302dfe721ad3ca5fb18d64c638f 2025-10-10T02:03:58.1310615Z deleted: sha256:e135fcbb872ffee1b10ef2430436bcdc6070f80bada4f6f29df4ff43ce22c5bf 2025-10-10T02:03:58.1311051Z deleted: sha256:b43b13192327ca33b41058e4ec301e99afe2545681946ef769a7a61cdcfbeca2 2025-10-10T02:03:58.1311486Z deleted: sha256:62563a12c0c16c746a7463f31b20031f14af18e2b07de16bed2dd6ae497cdda8 2025-10-10T02:03:58.1311930Z deleted: sha256:669aeb2091c6f16bf25899dca441b227b2f3f71c81b4f866752eea98afe33da0 2025-10-10T02:03:58.1312370Z deleted: sha256:662ed9daf3bbd7c75e1c6bbb332d7c26cbde354c93a15878b7e02beb0f34c678 2025-10-10T02:03:58.1312817Z deleted: sha256:6fde4e64ae77bf05993c494e6298fa4110b08631d69aab7a4772a15a8a442548 2025-10-10T02:03:58.1313257Z deleted: sha256:d9ead953554539f20688ecd6386499b2d82362b1b1c6b25021d0d7a69eec50e6 2025-10-10T02:03:58.1313693Z deleted: sha256:432a1c437c100c04fa7d6a8f39066ab56dde5a4861b64594b9c118d8b7b0675f 2025-10-10T02:03:58.1314129Z deleted: sha256:c30afe8220d7a2284c68b6064ac32cb056bfb4e4d40e3107107ce4599b5fec2e 2025-10-10T02:03:58.1314575Z deleted: sha256:88e9ee3ca3e2f5eec2be33087de509650ec96fdb43809c7230e0bb08750d7add 2025-10-10T02:03:58.1315011Z deleted: sha256:ca8d91823917321f63efd16926ca25c471fbd891b76c7b96783bb21013847838 2025-10-10T02:03:58.1315441Z deleted: sha256:e5757fc7a2da09412211d63b4fc54ce031023eae16fd5107165b5dd74ec3c6e7 2025-10-10T02:03:58.1315875Z deleted: sha256:908f42a23148aab74de493ebd7e97eea7865858e220b8aaa349f5d890479aa6e 2025-10-10T02:03:58.1316566Z deleted: sha256:c1e16b697497a8789e43bf5454ce8bdc219a4f1a0b4c684bb83098848349ee8c 2025-10-10T02:03:58.1317007Z deleted: sha256:6a220b588e3aa2555918233ab65a6abd91cb6764d772471c10c1c5d79f13cb72 2025-10-10T02:03:58.1317458Z deleted: sha256:ed65823401bbea17ab5c2b8ecc4e84c3d047fdf86c4044acd37cfe917849e506 2025-10-10T02:03:58.1317898Z deleted: sha256:a5b1e92804e52197e5e17c1f13907e802b3957d5f99011689965c433d58821f9 2025-10-10T02:03:58.1318334Z deleted: sha256:253e1d7fa0e015cdf247416952711f482e14604902a5d2b8cce9993a2a3b3d24 2025-10-10T02:03:58.1318851Z deleted: sha256:ca5ad77fb21d67464112b110aef81544ac5d43b2309cd12d61bed9bbfdc42e15 2025-10-10T02:03:58.1319411Z deleted: sha256:e94ee3c9b54243395ef87c3a1eec31ed7c6a31d37673b2d322816e08dbd1a118 2025-10-10T02:03:58.1319854Z deleted: sha256:89315ca20f065b6c143f279a913c3619669070f03be2bb6b2b03caef18998874 2025-10-10T02:03:58.1320289Z deleted: sha256:9aeae45b7bab4d904ee27a4e663b08e16e1be8e8a1c28c11328e8d666b482fe9 2025-10-10T02:03:58.1320802Z deleted: sha256:e52815a14f0f75252efae6743f2fce3814b200777535d74ab664e555e18886ff 2025-10-10T02:03:58.1321230Z deleted: sha256:e9f6ee51325b8f72704d3927dc62ccf70c72973e5c861b586404da7556ec4529 2025-10-10T02:03:58.1321659Z deleted: sha256:14132363c1ac9a5b1ab218a98d9209b5b5d523214bccbe6a9dae201d834208e8 2025-10-10T02:03:58.1322869Z deleted: sha256:4692931cca4fcefe71348bf3cffb47a7ceededdd131123f163f831d4bae1eac1 2025-10-10T02:03:58.1323726Z deleted: sha256:7ce9ee103794300058a597fc76e03268edc096ea2f1593770afaa73016f9f0e8 2025-10-10T02:03:58.1324199Z deleted: sha256:fb39599ef882320a257b5924b0a0ac99d8c62c3396cfe4508ff3d37556427421 2025-10-10T02:03:58.1324623Z deleted: sha256:8c676994914d959dc5ef16fe1d754c180a79c310513ebab96484cb97c28d99ad 2025-10-10T02:03:58.1325064Z deleted: sha256:d8d663334001dfecc055ccb2048c53a115763719fb55aaac728c5a5ec52e9354 2025-10-10T02:03:58.1325508Z deleted: sha256:cb9aec32a0cd5a45d9dae9fc89636bd5d1b7235a09352e85d88ed801a8c1a7bb 2025-10-10T02:03:58.1325959Z deleted: sha256:1bf54ca08eab0479e04daeb4450cd5359bda2139918d55576dbc435ef789c461 2025-10-10T02:03:58.1326386Z deleted: sha256:5d71c10e7be7172bf9b45248d8507e82c3119364e52c950940839da3984c4d5b 2025-10-10T02:03:58.1326808Z deleted: sha256:a9176111bd8471aa8d2d4f7d409a1547c59b74024372144b537a5f718350f74a 2025-10-10T02:03:58.1327252Z deleted: sha256:0946a9532ee6742ae519c31a401f2a013a94ddfe1bddcf6bff363a0c4910fe91 2025-10-10T02:03:58.1327705Z deleted: sha256:8e6353dc1ffcc20dda69135631eaa1aad42e0038e8db456b7d9d3eea4f111d75 2025-10-10T02:03:58.1328150Z deleted: sha256:b1b4a94efc01eaf3460de231b1685c299a122913e4e5df86b8b9d2a23d831437 2025-10-10T02:03:58.1328576Z deleted: sha256:2259025413435c0ff1a9a070523ae61e62c78530989b2c3330f0d23704a137b3 2025-10-10T02:03:58.1329007Z deleted: sha256:2acfa88bcefc179cc04eb4c414d36a47f7ec8444c41777613bbb3c0f2ccdaa5d 2025-10-10T02:03:58.1329458Z deleted: sha256:f98daf1fa3b1d8bcbef8cd4cea137f6d08fab572ecc5906c981e0dfe1dd76d42 2025-10-10T02:03:58.1329921Z deleted: sha256:3ffcf927608899854a2da49be3d0e41d77dd8d580b2b834338811fe9ad80fdc0 2025-10-10T02:03:58.1330356Z deleted: sha256:2059085fe5ac39f2b0398089556ef08128c7f87c71d9c6cc7940bf1e87de4b59 2025-10-10T02:03:58.1330788Z deleted: sha256:9ef72a5b6d2dfa7e843ab3735e33120bfcd1a9eaf5c602aebff0f9c4eb0984a8 2025-10-10T02:03:58.1331225Z deleted: sha256:90d3b2693815ff0e9a39f0733433a3043f62368ad99f513bc78adbaca85ffade 2025-10-10T02:03:58.1331679Z deleted: sha256:4bbbd89c196ee1904d74b95160dcbff15155c7824a4673698e1bf3c72b20873e 2025-10-10T02:03:58.1332403Z deleted: sha256:1a6b88de508df43a76a0243f7365abef52e51fa52b6e85a21cbb4d9b4844afe8 2025-10-10T02:03:58.1332853Z deleted: sha256:3080b6827f691d38c1ccada1ad766a71d4e605694b24f2a6264ac696b06d3952 2025-10-10T02:03:58.1333291Z deleted: sha256:c38bce02ca739cd758f8a6123addd2b5e45469761e91ac7f4391de364e02e87b 2025-10-10T02:03:58.1333730Z deleted: sha256:638cb10820f55ac2920396c01be87e60717f24fb04b4575eeaaf9e3da18169a1 2025-10-10T02:03:58.1334169Z deleted: sha256:fad077106111685ce7c6301a52c492f3533f691f213b2ad692821c52899b7405 2025-10-10T02:03:58.1334605Z deleted: sha256:6dbc7a719203586dc5dca848bff9ec50c54bb5478984eb9f4a1fa058e382da65 2025-10-10T02:03:58.1335033Z deleted: sha256:a33f103e4f676b2b0ef58e9428818918162447fb2bf885b60b0d1a2dd0814845 2025-10-10T02:03:58.1335470Z deleted: sha256:c7acd74e94da39045757348e6edb610ae80743b66fefb53fc5ba40746b6d620b 2025-10-10T02:03:58.1335915Z deleted: sha256:fc85ea9d45a7e990c9b3b2305376bd4b869b126167ff95a211eaef592b9595c5 2025-10-10T02:03:58.1336358Z deleted: sha256:01baeac660bbb677ba72b007e65d679ff1183c683767ef2ffb137a2c52804186 2025-10-10T02:03:58.1336878Z deleted: sha256:5af13118677cad47c318ab0e651bbbbe5745cb06de7e1edc1609bc8e64fc0b34 2025-10-10T02:03:58.1337317Z deleted: sha256:45a23f76e9d32ccc4cf19e7249e666bbff02ac5de3ac3e2d4da8d757d1fb2b91 2025-10-10T02:03:58.1337760Z deleted: sha256:767e56ba346ae714b6e6b816baa839051145ed78cfa0e4524a86cc287b0c4b00 2025-10-10T02:03:58.1338145Z untagged: public.ecr.aws/docker/library/python:3.13 2025-10-10T02:03:58.1339079Z untagged: public.ecr.aws/docker/library/python@sha256:4889af0e45f04b7c5dd741421a1280919499d38d3125d714b69fa86b23b1052a 2025-10-10T02:03:58.1339694Z deleted: sha256:6c82e3449d7794702180419555c0a0e1687ea79a0c665b250436286924681a55 2025-10-10T02:03:58.1340119Z deleted: sha256:68a0419cb3069ed43905ab41b911f2b7248601df62c854ae65e8c8a0342dbb30 2025-10-10T02:03:58.1340654Z deleted: sha256:b258354078ead7184c2f6d72eb3d5db1855162c0f80d164c09e794b21f30f48b 2025-10-10T02:03:58.1341119Z deleted: sha256:3e65e3c281dedcfdb54cb848bd29efd0e832cf5f29dec4b6b9849cd7420266cb 2025-10-10T02:03:58.1341569Z deleted: sha256:42f4cd5b256627f333ad4537462aac85c359e741da4f02d1cb68600c128841c5 2025-10-10T02:03:58.1341999Z deleted: sha256:4e7df8e345c749980c75fd48e7b2ef15e63dc912b467ffa446284f0dbcc5aa33 2025-10-10T02:03:58.1342428Z deleted: sha256:345f9c4d6fe93d61688b6f1a607137261d7983d3788b5d88e8791b6ebeb8a920 2025-10-10T02:03:58.1342865Z deleted: sha256:a5ec5ec9d16c5551ce8889cbc03af0609b92cf8a8d60b32e72a7eabb8378eaec 2025-10-10T02:03:58.1343132Z 2025-10-10T02:03:58.1364221Z Total reclaimed space: 45.73GB 2025-10-10T02:03:58.1423054Z ##[group]Run set +e 2025-10-10T02:03:58.1423323Z set +e 2025-10-10T02:03:58.1423505Z set -x 2025-10-10T02:03:58.1423666Z  2025-10-10T02:03:58.1423821Z nvidia-smi 2025-10-10T02:03:58.1424170Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2025-10-10T02:03:58.1424697Z # the case where the driver has already crashed as it still can get the driver version 2025-10-10T02:03:58.1425172Z # and some basic information like the bus ID. However, the rest of the information 2025-10-10T02:03:58.1425540Z # would be missing (ERR!), for example: 2025-10-10T02:03:58.1425772Z # 2025-10-10T02:03:58.1425992Z # +-----------------------------------------------------------------------------+ 2025-10-10T02:03:58.1426382Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2025-10-10T02:03:58.1426784Z # |-------------------------------+----------------------+----------------------+ 2025-10-10T02:03:58.1427163Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-10-10T02:03:58.1427563Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2025-10-10T02:03:58.1427896Z # | | | MIG M. | 2025-10-10T02:03:58.1428157Z # |===============================+======================+======================| 2025-10-10T02:03:58.1428449Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2025-10-10T02:03:58.1428776Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2025-10-10T02:03:58.1429084Z # | | | ERR! | 2025-10-10T02:03:58.1429395Z # +-------------------------------+----------------------+----------------------+ 2025-10-10T02:03:58.1429660Z # 2025-10-10T02:03:58.1429873Z # +-----------------------------------------------------------------------------+ 2025-10-10T02:03:58.1430198Z # | Processes: | 2025-10-10T02:03:58.1430529Z # | GPU GI CI PID Type Process name GPU Memory | 2025-10-10T02:03:58.1430834Z # | ID ID Usage | 2025-10-10T02:03:58.1431228Z # |=============================================================================| 2025-10-10T02:03:58.1431523Z # +-----------------------------------------------------------------------------+ 2025-10-10T02:03:58.1431782Z # 2025-10-10T02:03:58.1432047Z # This should be reported as a failure instead as it will guarantee to fail when 2025-10-10T02:03:58.1432495Z # Docker tries to run with --gpus all 2025-10-10T02:03:58.1432725Z # 2025-10-10T02:03:58.1432989Z # So, the correct check here is to query one of the missing piece of info like 2025-10-10T02:03:58.1433369Z # GPU name, so that the command can fail accordingly 2025-10-10T02:03:58.1433716Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-10-10T02:03:58.1434026Z NVIDIA_SMI_STATUS=$? 2025-10-10T02:03:58.1434215Z  2025-10-10T02:03:58.1434521Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-10-10T02:03:58.1434983Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-10-10T02:03:58.1435397Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-10-10T02:03:58.1435752Z  .github/scripts/stop_runner_service.sh 2025-10-10T02:03:58.1435982Z fi 2025-10-10T02:03:58.1436146Z  2025-10-10T02:03:58.1436540Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2025-10-10T02:03:58.1436999Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2025-10-10T02:03:58.1437380Z # https://github.com/pytorch/test-infra/issues/4000 2025-10-10T02:03:58.1437693Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2025-10-10T02:03:58.1437960Z NVIDIA_SMI_STATUS=$? 2025-10-10T02:03:58.1438147Z  2025-10-10T02:03:58.1438456Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-10-10T02:03:58.1438921Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-10-10T02:03:58.1439463Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-10-10T02:03:58.1439814Z  .github/scripts/stop_runner_service.sh 2025-10-10T02:03:58.1440063Z fi 2025-10-10T02:03:58.1440213Z  2025-10-10T02:03:58.1440386Z # Check the GPU count to be a power of 2 2025-10-10T02:03:58.1440791Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2025-10-10T02:03:58.1441315Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2025-10-10T02:03:58.1441708Z  .github/scripts/stop_runner_service.sh 2025-10-10T02:03:58.1441934Z fi 2025-10-10T02:03:58.1453819Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:03:58.1454098Z env: 2025-10-10T02:03:58.1454255Z GIT_DEFAULT_BRANCH: main 2025-10-10T02:03:58.1454497Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-10-10T02:03:58.1454911Z DOCKER_CONTAINER_ID: 3e581cf95cadf79e78abe6e7ca8fedc542b29bd0369c8642fc9936d124129f6f 2025-10-10T02:03:58.1455278Z ##[endgroup] 2025-10-10T02:03:58.1489150Z + nvidia-smi 2025-10-10T02:03:58.1751046Z Fri Oct 10 02:03:58 2025 2025-10-10T02:03:58.1751426Z +-----------------------------------------------------------------------------------------+ 2025-10-10T02:03:58.1751902Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-10-10T02:03:58.1752361Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T02:03:58.1752821Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-10-10T02:03:58.1753450Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-10-10T02:03:58.1753857Z | | | MIG M. | 2025-10-10T02:03:58.1754144Z |=========================================+========================+======================| 2025-10-10T02:03:58.1870086Z | 0 NVIDIA L4 On | 00000000:35:00.0 Off | 0 | 2025-10-10T02:03:58.1870772Z | N/A 34C P8 16W / 72W | 0MiB / 23034MiB | 0% Default | 2025-10-10T02:03:58.1871209Z | | | N/A | 2025-10-10T02:03:58.1871618Z +-----------------------------------------+------------------------+----------------------+ 2025-10-10T02:03:58.1873583Z 2025-10-10T02:03:58.1873769Z +-----------------------------------------------------------------------------------------+ 2025-10-10T02:03:58.1874183Z | Processes: | 2025-10-10T02:03:58.1874603Z | GPU GI CI PID Type Process name GPU Memory | 2025-10-10T02:03:58.1874972Z | ID ID Usage | 2025-10-10T02:03:58.1875275Z |=========================================================================================| 2025-10-10T02:03:58.1878905Z | No running processes found | 2025-10-10T02:03:58.1879470Z +-----------------------------------------------------------------------------------------+ 2025-10-10T02:03:58.4133443Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-10-10T02:03:58.4292209Z NVIDIA L4 2025-10-10T02:03:58.4326498Z + NVIDIA_SMI_STATUS=0 2025-10-10T02:03:58.4326781Z + '[' 0 -ne 0 ']' 2025-10-10T02:03:58.4333260Z ++ nvidia-smi --list-gpus 2025-10-10T02:03:58.4334787Z ++ wc -l 2025-10-10T02:03:58.4524701Z + GPU_COUNT=1 2025-10-10T02:03:58.4524952Z + NVIDIA_SMI_STATUS=0 2025-10-10T02:03:58.4525184Z + '[' 0 -ne 0 ']' 2025-10-10T02:03:58.4525391Z + '[' 1 -le 8 ']' 2025-10-10T02:03:58.4525600Z + '[' 1 -ne 1 ']' 2025-10-10T02:03:58.4612801Z Post job cleanup. 2025-10-10T02:03:58.4657453Z Post job cleanup. 2025-10-10T02:03:58.5593845Z [command]/usr/bin/git version 2025-10-10T02:03:58.5654192Z git version 2.50.1 2025-10-10T02:03:58.5687035Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/4e4d2d83-8272-432d-91c7-ed62f4e12dbe/.gitconfig' 2025-10-10T02:03:58.5696542Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4e4d2d83-8272-432d-91c7-ed62f4e12dbe' before making global git config changes 2025-10-10T02:03:58.5697245Z Adding repository directory to the temporary git global config as a safe directory 2025-10-10T02:03:58.5701487Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T02:03:58.5739178Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-10-10T02:03:58.5772815Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-10-10T02:03:58.6156867Z Entering 'android/libs/fbjni' 2025-10-10T02:03:58.6228032Z Entering 'third_party/FP16' 2025-10-10T02:03:58.6302254Z Entering 'third_party/FXdiv' 2025-10-10T02:03:58.6373527Z Entering 'third_party/NNPACK' 2025-10-10T02:03:58.6444622Z Entering 'third_party/NVTX' 2025-10-10T02:03:58.6515028Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T02:03:58.6584471Z Entering 'third_party/XNNPACK' 2025-10-10T02:03:58.6668117Z Entering 'third_party/aiter' 2025-10-10T02:03:58.6739808Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T02:03:58.6820893Z Entering 'third_party/benchmark' 2025-10-10T02:03:58.6892762Z Entering 'third_party/composable_kernel' 2025-10-10T02:03:58.6973348Z Entering 'third_party/cpp-httplib' 2025-10-10T02:03:58.7044534Z Entering 'third_party/cpuinfo' 2025-10-10T02:03:58.7116657Z Entering 'third_party/cudnn_frontend' 2025-10-10T02:03:58.7190889Z Entering 'third_party/cutlass' 2025-10-10T02:03:58.7270979Z Entering 'third_party/fbgemm' 2025-10-10T02:03:58.7344403Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T02:03:58.7415063Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T02:03:58.7490488Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T02:03:58.7565783Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T02:03:58.7643656Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T02:03:58.7714421Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T02:03:58.7783303Z Entering 'third_party/fbgemm/external/json' 2025-10-10T02:03:58.7857097Z Entering 'third_party/flash-attention' 2025-10-10T02:03:58.7928534Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T02:03:58.8005204Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T02:03:58.8084686Z Entering 'third_party/flatbuffers' 2025-10-10T02:03:58.8156772Z Entering 'third_party/fmt' 2025-10-10T02:03:58.8226023Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T02:03:58.8294457Z Entering 'third_party/gloo' 2025-10-10T02:03:58.8364682Z Entering 'third_party/googletest' 2025-10-10T02:03:58.8434315Z Entering 'third_party/ideep' 2025-10-10T02:03:58.8505466Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T02:03:58.8581960Z Entering 'third_party/ittapi' 2025-10-10T02:03:58.8654013Z Entering 'third_party/kineto' 2025-10-10T02:03:58.8723102Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T02:03:58.8790255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T02:03:58.8863863Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T02:03:58.8934357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T02:03:58.9005510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T02:03:58.9072664Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T02:03:58.9148124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T02:03:58.9223257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T02:03:58.9294445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T02:03:58.9366121Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T02:03:58.9435058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T02:03:58.9504173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T02:03:58.9576771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T02:03:58.9656086Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T02:03:58.9725472Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T02:03:58.9797299Z Entering 'third_party/kleidiai' 2025-10-10T02:03:58.9871660Z Entering 'third_party/mimalloc' 2025-10-10T02:03:58.9942639Z Entering 'third_party/nlohmann' 2025-10-10T02:03:59.0015514Z Entering 'third_party/onnx' 2025-10-10T02:03:59.0100301Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T02:03:59.0177318Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T02:03:59.0250368Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T02:03:59.0322896Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T02:03:59.0393355Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T02:03:59.0463559Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T02:03:59.0534087Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T02:03:59.0604920Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T02:03:59.0674294Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T02:03:59.0742824Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T02:03:59.0816091Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T02:03:59.0890122Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T02:03:59.0983570Z Entering 'third_party/pocketfft' 2025-10-10T02:03:59.1054405Z Entering 'third_party/protobuf' 2025-10-10T02:03:59.1125423Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T02:03:59.1194580Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T02:03:59.1267454Z Entering 'third_party/psimd' 2025-10-10T02:03:59.1345009Z Entering 'third_party/pthreadpool' 2025-10-10T02:03:59.1414612Z Entering 'third_party/pybind11' 2025-10-10T02:03:59.1484933Z Entering 'third_party/python-peachpy' 2025-10-10T02:03:59.1559921Z Entering 'third_party/sleef' 2025-10-10T02:03:59.1632229Z Entering 'third_party/tensorpipe' 2025-10-10T02:03:59.1703154Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T02:03:59.1773372Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T02:03:59.1842534Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T02:03:59.1912696Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T02:03:59.1980841Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T02:03:59.2077051Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-10-10T02:03:59.2100849Z http.https://github.com/.extraheader 2025-10-10T02:03:59.2110420Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-10-10T02:03:59.2144214Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-10-10T02:03:59.2509718Z Entering 'android/libs/fbjni' 2025-10-10T02:03:59.2558746Z http.https://github.com/.extraheader 2025-10-10T02:03:59.2604146Z Entering 'third_party/FP16' 2025-10-10T02:03:59.2650844Z http.https://github.com/.extraheader 2025-10-10T02:03:59.2694477Z Entering 'third_party/FXdiv' 2025-10-10T02:03:59.2740008Z http.https://github.com/.extraheader 2025-10-10T02:03:59.2783953Z Entering 'third_party/NNPACK' 2025-10-10T02:03:59.2830992Z http.https://github.com/.extraheader 2025-10-10T02:03:59.2876147Z Entering 'third_party/NVTX' 2025-10-10T02:03:59.2922790Z http.https://github.com/.extraheader 2025-10-10T02:03:59.2966301Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T02:03:59.3011901Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3055282Z Entering 'third_party/XNNPACK' 2025-10-10T02:03:59.3100850Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3158850Z Entering 'third_party/aiter' 2025-10-10T02:03:59.3207962Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3252224Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T02:03:59.3299738Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3353750Z Entering 'third_party/benchmark' 2025-10-10T02:03:59.3401598Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3445060Z Entering 'third_party/composable_kernel' 2025-10-10T02:03:59.3490663Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3541428Z Entering 'third_party/cpp-httplib' 2025-10-10T02:03:59.3588106Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3632324Z Entering 'third_party/cpuinfo' 2025-10-10T02:03:59.3678643Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3723653Z Entering 'third_party/cudnn_frontend' 2025-10-10T02:03:59.3769896Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3814035Z Entering 'third_party/cutlass' 2025-10-10T02:03:59.3860265Z http.https://github.com/.extraheader 2025-10-10T02:03:59.3913542Z Entering 'third_party/fbgemm' 2025-10-10T02:03:59.3960067Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4006668Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T02:03:59.4051246Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4094513Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T02:03:59.4140214Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4191887Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T02:03:59.4239255Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4283959Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T02:03:59.4329267Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4382947Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T02:03:59.4429498Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4473543Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T02:03:59.4519942Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4564235Z Entering 'third_party/fbgemm/external/json' 2025-10-10T02:03:59.4610493Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4659249Z Entering 'third_party/flash-attention' 2025-10-10T02:03:59.4708038Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4752713Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T02:03:59.4800744Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4849770Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T02:03:59.4897925Z http.https://github.com/.extraheader 2025-10-10T02:03:59.4954464Z Entering 'third_party/flatbuffers' 2025-10-10T02:03:59.5003574Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5051696Z Entering 'third_party/fmt' 2025-10-10T02:03:59.5103331Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5147351Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T02:03:59.5197919Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5243677Z Entering 'third_party/gloo' 2025-10-10T02:03:59.5290145Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5334119Z Entering 'third_party/googletest' 2025-10-10T02:03:59.5381484Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5427072Z Entering 'third_party/ideep' 2025-10-10T02:03:59.5472505Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5514619Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T02:03:59.5560599Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5612548Z Entering 'third_party/ittapi' 2025-10-10T02:03:59.5659540Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5705505Z Entering 'third_party/kineto' 2025-10-10T02:03:59.5752030Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5795020Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T02:03:59.5841561Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5885191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T02:03:59.5931501Z http.https://github.com/.extraheader 2025-10-10T02:03:59.5977443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T02:03:59.6023537Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6067570Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T02:03:59.6115447Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6160595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T02:03:59.6209349Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6253018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T02:03:59.6300594Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6348437Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T02:03:59.6397826Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6441258Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T02:03:59.6487424Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6536018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T02:03:59.6581126Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6628769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T02:03:59.6677331Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6724292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T02:03:59.6770824Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6815200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T02:03:59.6861625Z http.https://github.com/.extraheader 2025-10-10T02:03:59.6908048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T02:03:59.6957302Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7009019Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T02:03:59.7057179Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7103367Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T02:03:59.7150179Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7198505Z Entering 'third_party/kleidiai' 2025-10-10T02:03:59.7247188Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7292824Z Entering 'third_party/mimalloc' 2025-10-10T02:03:59.7338599Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7382964Z Entering 'third_party/nlohmann' 2025-10-10T02:03:59.7430456Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7475718Z Entering 'third_party/onnx' 2025-10-10T02:03:59.7522028Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7581787Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T02:03:59.7628975Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7678659Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T02:03:59.7728846Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7773886Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T02:03:59.7819474Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7864115Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T02:03:59.7909454Z http.https://github.com/.extraheader 2025-10-10T02:03:59.7954491Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T02:03:59.7999831Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8043447Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T02:03:59.8089694Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8135122Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T02:03:59.8180721Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8224368Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T02:03:59.8267336Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8312137Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T02:03:59.8358885Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8403803Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T02:03:59.8449149Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8496882Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T02:03:59.8542432Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8589268Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T02:03:59.8637106Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8703836Z Entering 'third_party/pocketfft' 2025-10-10T02:03:59.8749438Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8794061Z Entering 'third_party/protobuf' 2025-10-10T02:03:59.8841418Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8887381Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T02:03:59.8935060Z http.https://github.com/.extraheader 2025-10-10T02:03:59.8980751Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T02:03:59.9028694Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9076609Z Entering 'third_party/psimd' 2025-10-10T02:03:59.9123050Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9166477Z Entering 'third_party/pthreadpool' 2025-10-10T02:03:59.9213904Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9256749Z Entering 'third_party/pybind11' 2025-10-10T02:03:59.9304520Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9347738Z Entering 'third_party/python-peachpy' 2025-10-10T02:03:59.9395261Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9439473Z Entering 'third_party/sleef' 2025-10-10T02:03:59.9486464Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9532761Z Entering 'third_party/tensorpipe' 2025-10-10T02:03:59.9580352Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9623571Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T02:03:59.9670090Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9714929Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T02:03:59.9761926Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9804951Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T02:03:59.9851098Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9895296Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T02:03:59.9940564Z http.https://github.com/.extraheader 2025-10-10T02:03:59.9983251Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T02:04:00.0030654Z http.https://github.com/.extraheader 2025-10-10T02:04:00.0178644Z A job completed hook has been configured by the self-hosted runner administrator 2025-10-10T02:04:00.0204448Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-10-10T02:04:00.0211264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T02:04:00.0211551Z ##[endgroup] 2025-10-10T02:04:07.3862141Z Cleaning up orphan processes